Paper notes ACL 2022 unified structure generation for universal information extraction

1 brief introduction

Thesis title ：Unified Structure Generation for Universal Information Extraction
Source of the paper ：ACL 2022
Organization ： Software Institute Baidu
Thesis link ：https://arxiv.org/pdf/2203.12277.pdf
Code link ：https://github.com/universal-ie/UIE

1.1 motivation

• The task specific information extraction methods hinder the structural development of information extraction systems 、 Knowledge sharing and cross domain migration .

1.2 innovation

• A unified text-to-structure Generate schema , Different information can be extracted (IE) Task modeling , Generate the target structure adaptively , And learn general information extraction ability from different knowledge resources . Is the first text-to-structure Pre training extraction model .
• A unified structure generation network is designed , Extracting languages from structures (structural extraction language) The heterogeneous information extraction structure is encoded into a unified representation , And through the structural model (structural schema instructor) Guiding mechanism control UIE Model recognition 、 Relate and generate .
  

2 Method

The overall framework of the model is shown in the figure below , It mainly includes structural schema instructor and structural extraction language Two parts , Given a specific predefined schema s And the text t, The model needs to generate a structure , The structure contains schema s Indicated text t Structure information required in .

2.1 Structured Extraction Language

structured exextraction language (SEL) Will be heterogeneous IE The structure is encoded as a unified representation , There are three semantic structures , An example is shown below ：

1. SPOTNAME： Indicates that the... Exists in the text Spot Name Information fragment of type ;
2. ASSONAME： It indicates that there is an upper layer in the text and structure Spot Yes Asso Name Pieces of information about the relationship ;
3. INFOSPAN： Express Spot Name perhaps Asso Name In the text span;
   

2.2 Structural Schema Instructor

Structural Schema Instructor(SSI) Describe the extraction objectives of the task , Construct a schema-based prompt. Contains three types of token：

1. SPOTNAME： Target spot name.
2. ASSONAME： Target association name.
3. Special Symbols([spot], [asso],[text])： Add to each spot name、association name And before the text .
   

2.3 Structure Generation with UIE

text-to-SEL The generated process uses encoding - Decoding structure , The structure is Transformer, The encoding and decoding formulas are as follows ：

3 Pre-training and Fine-tuning for UIE

3.1 Pre-training

UIE Need encoded text 、 Map text to structure 、 Decoding structure , The data set of pre training includes three types

1. $D_{pair}$： Text - A parallel corpus of structures , Each data includes token Sequence x And structural records y, Pre trained text to structure mapping ability (UIE), Some negative samples were randomly sampled during pre training (spots、association),loss The formula is as follows ：
   
2. $D_{record}$： Structural corpus , Pre training the ability to generate structures ( decoder ),loss The formula is as follows ：
   
3. $D_{text}$： Unstructured text corpus , Use masked language model Way to pre train semantic representation , loss The formula is as follows ：
   
   total loss The formula is as follows , At every batch Randomly select data for different tasks in .
   

3.2 On-Demand Fine-tuning

UIE Fine tune for different dirty tasks ,$D_{task}=(s,x,y)$,loss by teacher-forcing Cross entropy , To mitigate exposure bias , Set up Rejection Mechanism, Insert some randomly [NULL] Node as a negative example SPOTNAME and ASSONAME, Here's the picture

4 experiment

The supervised experimental results are shown in the figure below ：

The experimental results under low resources are shown in the figure below ：

Ablation Experiment ：