Uie: unified model of information extraction

Thesis link : https://arxiv.org/abs/2203.12277

Part1 background

Recently due to business needs , I have been paying attention to some articles in the field of information extraction , I tried in the experiment BERT+Softmax、BERT+NER as well as GlobalPointer Wait for the model , The effect was pretty good , It's just that the standard data is a little expensive . therefore , Looking for some few-shot A better model , Auxiliary annotation . Unintentionally , I found this paper , Try it zero-shot experiment , The effect is amazing .

as everyone knows , Information extraction usually includes four common sub tasks : Entity extraction 、 Relationship extraction 、 Event extraction and emotion analysis . in the past , Because different tasks recognize entities 、 The types of events are different , Therefore, specific models should be trained for specific tasks , High customization , Not universal . In response to this problem , This paper considers the problem of information extraction from the perspective of generative model , So that these sub tasks can be completed through a model .

Now let's take a look at the specific parts of the model .

Part2 Model

1SEL(Structured Extraction Language)

The labels of information extraction tasks are various , Yes SBME The way , There are also directly expressed by the starting and ending positions . In order to unify modeling , Need to be different IE(information extraction) Task tags are encoded in a unified form . Therefore, a structured extraction language is proposed SEL.

IE In fact, the structure can be summarized into two atomic operations :

• Spotting: It refers to the segment that locates the target information in the sentence , Such as the entity or trigger word in the event .
• Associating: It represents the relationship between two different pieces of information . for example , The relationship between entity pairs or the role of events .

for instance :

Suppose the input sentence is :"Steve became CEO of Apple in 1997.", There are three entities , people : Steve、 company :Apple、 Time : 1997. The following figure shows the representation of extraction structure in structured extraction language . For example, blue represents relationship extraction , Red represents the event extraction , Finally, the extraction of entities . The trigger word in the red part is became. Employer is Apple, Employee is Steve, Time in 1997 year .

It can be seen that SEL The advantages of :

1. For different IE Structure is uniformly coded , Therefore, different IE Tasks are modeled as the same text-to-structure The generation process ;
2. Effectively express all the extracted results of a sentence as the same structure , Can naturally carry out joint extraction ;
3. The resulting output structure is very compact , The complexity of decoding is greatly reduced .

2SSI(Structural Schema Instructor)

Use SEL, UIE( The model proposed in this paper ) Different... Can be generated evenly IE structure . However , Because of different IE Tasks have different modes , In the extraction process , How to adaptively control the information we want to generate ？ This paper presents a structured pattern guidance , Be similar to Prompt. Put the relationship types that need to be extracted 、 Entity type, etc. are spliced with sentences . Here's the picture :

The extraction type of a given structured pattern S And the sequence of text X, adopt UIE The model can generate SEL Structured information . namely :

The specific input can be expressed as :

Actually, it means the above figure , Splice the type to be extracted with the text , Input to model . This may involve the defects of this article , If the extracted entity type 、 If there are many relationship types , The length of input text may be very long , Efficiency is a big problem ..

3UIE( Universal Information Extraction)

So much has been said before , It's still cloudy , What is a model ？ In fact, it is a standard Transformer, Contains Encoder and Decoder. First of all, will SSI Information and sentence splicing , use encoder De coding , As shown below :

among yes SSI Information , It's a sentence .

next , The extracted information is decoded by autoregression . As shown below :

Read here , Some people may feel that this method is a little lame , The generation task is uncontrollable , If the generated information structure does not conform to the structure defined above , How to extract information ？ The author adopted a different definition of loss to avoid this situation .

4loss Design

Text-to-Structure Pre-training using

First , Definition . Specifically x Input is :[spot] person [asso] work for [text]Steve became CEO of Apple in 1997.,y Then for :((person: Steve(work for: Apple))). It can be found that during the generation process ,person and work for It must appear when outputting . These two things are what we defined before spotting、associating. The author found that if in the generated token in , Add a loss , Used to judge the current token Is it right? spotting Or not associating The effect will be better . The positive sample here is spotting perhaps assocating, Negative samples are randomly selected token. The losses are as follows :

Structure Generation Pre-training

This loss is easy to understand , In the generation task , A loss of autoregression . As shown below :

Retrofitting Semantic Representation

In order to improve the UIE The semantic representation of , The author also added MLM Mission . The losses are as follows :

there MASK Is for the target text . Follow bert A little difference between .

Finally, add these three losses , Conduct large-scale pre training . thus , That's all for the model . Finally, let's look at the experiment .

Part3 experiment

In this paper 13 individual IE Experiments were carried out on the benchmark , involve 4 A very representative IE Mission ( Including entity extraction 、 Relationship extraction 、 Event extraction 、 Structured emotion extraction ) And its combination ( for example , Joint entities - Relationship extraction ). give the result as follows :

among SEL For those without pre training UIE Model , That is to say, directly use T5 Do this generation , It can be found that the effect is also very good . A unified generation model is just around the corner . Plus pre training , The effect has also been improved to varying degrees .

in addition , The author also gives in addition to few-shot The effect of , as follows :

It's a little amazing ....