Raki's notes on reading paper: Leveraging type descriptions for zero shot named entity recognition and classification

Abstract & Introduction & Related Work

• Research tasks
  Named entity recognition and classification

• Existing methods and related work

• Facing the challenge

• Innovative ideas

  1. The first method for zero-shot Of NERC Methods
  2. Introduce a new architecture , Use the text of many entity categories to describe naturally occurring facts

• The experimental conclusion

The non entity words observed during the training may belong to one of the test categories , Based on this observation, we propose several different architectures , Based on the cross attention, we can use... Between statement and type description transformer, Combined with the pre training model

We explore modeling negative classes in the following ways

1. Use the description of negative classes
2. Directly establish the model of negative class
3. Model negative classes using the representations generated for the class corresponding to the type
   

Zero-shot NERC

For each type d, The cross attention coder generates a vector representation v, For each in statement s The words in w

Make a linear transformation ,o Means the word t How likely are they to belong to entity classes c

In order to be able to do more than just classify entities , And identify them , every last token The score of is spliced with the score belonging to the negative category , Corresponds to not belonging to any of the types considered ：

Choose the category with the highest score

Cross-attention Encoder

use bert

Modelling the negative class

Just like No 1 What is discussed in section , Non entity classes create a challenging setup . The non entity words observed during the training may belong to one of the test classes .

Description-based encoding

However , This approach requires the description of something other than . This makes it difficult to make a wise decision on the most appropriate description in practice . in addition , Non physical markers are likely to differ in training and testing , So a fixed description is unlikely to perform well

Independent encoding

Negative classes can be modeled directly , Because it is observed in the training data . therefore , Without considering any description , Each tag is represented as a negative class in the context of the sentence , Instead of exploring cross attention

Class-aware encoding

Description based coding and independent coding do not model the following facts , That is to say zero-shot Of NERC in , Not every entity marked as non entity during training is non entity during testing . contrary , We propose to model negative classes by combining the representations of other classes generated by the cross attention coder ：$v_{t,c_0},...,v_{t,c_k}$. then , Each vector is transformed linearly , Use $w_{neg-cl}$, Then concatenate them into a characteristic graph m

Training

To prevent the attention coder from over fitting a small number of category descriptions , We use a regularizer in the form of an entity mask , This regularization avoids vocabulary memory , The model is encouraged to learn the relationship between entity context and category description , While still learning to incorporate aspects of the entity itself （ For example, capital letters 、 shape 、 form ）, And relate them to type descriptions

Because negative labels are not balanced , Use

Evaluation setup

Experiments

Conclusions & Future work

This paper discusses zero point shooting with entity type description NERC The task of , Transfer knowledge from observed classes to unseen classes . We propose a multi class architecture to solve the problem of zero shooting NERC Specific challenges , This architecture uses class aware coding to model negative classes , Thus, the definition of non entity classes is not clear . These models are based on OntoNotes and MedMentions The zero adaptation of the data set was evaluated . Results show , The proposed model is superior to the powerful baseline , And further shows that high-quality entity description （ Note guide ） It is an effective way to transfer knowledge from observed to unseen classes . Future work will aim to incorporate the predicted dependencies between tags

Remark

This article is really about ,NER Just NER, I have to say something NERC, Attention is attention , I have to say cross attention , You seem to be novel Is it? ？ I'm speechless