Voice assistant - Qu - ner and intention slot model

        This chapter , Let's take a look at the voice assistant NER And intention slot model , It is divided into the following parts ：1、 What is? NER？2、 What is the intention slot model ？ What design forms can there be ？3、 frequently-used NER And what are the intended slot models ？

One 、 What is? NER

        NER It's called named entity recognition , That is to extract the entities in a sentence , Entities can be time 、 place 、 figure 、 Music name 、 TV play title, etc , Take the following example ： it's a nice day today , The entities contained therein are ： Time ：“ today ”.

        NER The essence of the model is a sequence annotation task , That is, each word in a paragraph of text is labeled with a corresponding label , The commonly used label is ：B( Entity start ),M( In the middle of the entity ), E( End of entity ),O( Non entity ), S( Single entity , That is, the word is an independent entity ), Take the following example ： today （B-TIME） God (E-TIME) God (O) gas (O) No (O) wrong (O), Of course, here it is “ Entity ” It needs to be artificially defined , If you only care about the weather entities , Then the above example only needs to extract “ today ” That's all right. , If you still care about the weather type , The weather type is also an entity , such as ： today （B-TIME） God (E-TIME) Meeting (O) Next (B-WEATHER_TYPE) rain (E-WEATHER_TYPE) Do you (O).

Two 、 What is the intention slot model

        In the voice assistant , You need to understand the user's intention in one sentence , And extract the concerned keywords （ Slot position ）, such as ：“ Will it rain today ”, In this sentence Intention by ： Weather query ; Slot position by ： Time ：” today “; Weather type ：” It's raining “. According to the intention and slot information , The assistant needs to check whether the current weather will rain , Then reply to the user .

        For this task , There are two ways to design ：1、 Use the two models to do the task separately , Classification +NER, Use classification to identify query Intention , utilize NER To extract the slot position .2、 Think of the task as two tasks , But use a model to do , Joint learning , Learning at the same time query Intent and slot information in . We generally recommend using 2 To do it , For the following reasons ：

1. For one query, The intention and slot position information interact with each other , That is, the intention information will affect the marking of slots , Slot information will in turn affect the identification of intention . And the intention and slot position are also used together in real use , If the two don't correspond , It may lead to incorrect skill execution . for instance ：query1： Play the song grandma's house . query2： Navigation to grandma's house for dinner . here query1 Intended to play music , here ” Grandma's Home “ Name the song ,query2 The intention is to navigate , here ” Grandma's Home “ For the location .
2. If you split it into two models , The final effect is the combination of the two models , If the accuracy of a model is 90%, The other is for 80%, The final effect is 72%, According to the experimental comparison , Although joint learning programs are classified and NER The sum of the two tasks may be slightly less effective than the simple two models , But the comprehensive effect is stronger than the splitting of the two models .

3、 ... and 、 Common model design schemes

        First , Confirm the input and output of the model , In the prediction phase , Enter a sentence query, The output is intended +NER result . In the training phase , Belong into a sentence query, And intention information and NER Label information , such as ：query：” it's a nice day today “.intent:"weather_forecast",slot：”time:[( today ,0,1)],weather_type:[( The weather ,2,3)]“, here slot Explain in the form of data ： Entity type :[( Entity , The beginning of the entity , Entity end position )]. In actual training , Will slot The data is formatted as ：B-TIME, E-TIME,B-WEATHER_TYPE,E-WEATHER_TYPE,O,O

        Then let's look at the design of the model , For simple NER Model , The model schemes that can be used are ：BILSTM-CRF, The two-way lstm+CRF, Another scheme is ：transformer（encoder part ）+CRF, utilize CRF（ Conditional random field ） Can greatly improve NER The accuracy of the entity , This is in NER Is already a defined module （ Of course , If you use BERT,CRF It's not necessary , however BERT It's too big , Its speed and deployment cost are generally difficult to meet online requirements ）, These two schemes are both mature schemes , No more details here , The effect is similar .

        Next, let's look at the model design of intentional slot joint learning , There are two designs ：1、transformer（encoder part ）+CRF.2、SF-ID. Let's take a look .

programme 1：ransformer（encoder part ）+CRF：

         Pictured above , take query Encoded as embedding after , Pass a layer transformer-encoder code , This part serves as a shared layer in joint tasks , Next, two decoding layers are made for intention recognition and slot extraction , among intent Some use a layer transformer-encoder+FC To make a classification ,slot Partial utilization transformer-encoder+FC+CRF To do it NER Mission .

        Maybe some students will ask , Do it here slot Why can't the decoding part of the task be used transformer-decoder Part of it ？ do NER In less than transformer The decoder of , Just use the encoder to get the context representation of each word ,transformer-decoder Part is generally used for generating tasks , Use the generated sequence and input sequence to do attention, The generated sequence does self-attention, Get the next word output . Of course , Do it here NER It is not impossible to use transformer-decoder, It's just not necessary , This task is relatively simple , utilize CRF It's already pretty good .

programme 2：SF-ID：

        The overall structure of the scheme is shown in the figure below ：

The corresponding paper information is as follows ：

• A Novel Bi-directional Interrelated Model for Joint Intent Detection and Slot Filling
• Source of the article ：ACL2019
• Download address ：A Novel Bi-directional Interrelated Model for Joint Intent Detection and Slot Filling
• Paper source code (tf)：https://github.com/ZephyrChenzf

This method proposes a two-way correlation model to complete SLU Medium slot filling and intent detection Mission , The model can provide effective information between two tasks to improve performance （iteration mechanism）, See this article for a detailed analysis ：SLU And slot filling and intent detection（2）- Reading notes - You know  , It's not going to unfold here .

Four 、 Some optimization schemes

1、 The effect of the model is improved ：

        Improve the model effect , Distillation can be used , It is using bert Distillation transformer.

2、 The recognition effect of unlisted words ：

        Problem description ： For some words that do not appear in the training set ,NER May identify errors , In particular, some new words appear frequently and have no obvious features , Like music , Music names are frequently updated and have no fixed features , Therefore, the recognition effect of some new song models that do not appear may be poor .

        Solution ： For this question , Can pass slot_dict To solve the problem , The idea is ： Put new words in a dictionary , Train a dictionary of new words word_embedding, Every time the model is used NER When identifying , Look it up in the dictionary , For the words found in the original embedding Layer plus the found word_embedding Enhance the weight of the corresponding position to improve its characteristics . This scheme can be realized by updating the dictionary of new words only word_embedding, It is not necessary to retrain the whole model to improve the recognition effect of new words .