Reading of the paper "attentional encoder network for targeted sentimental classification"

Article address ：https://arxiv.org/pdf/1902.09314.pdf

Article content

   Previously, emotional classification based on specific aspects of entities is mostly used RNN And attention mechanism , However RNN Henan parallelization , And when the sentence is too long, it also brings difficulties to long-term memory . Therefore, this question proposes an attention coding network AEN（Attentional Encoder Network）, To model context and target entities . And the paper also raises the problem of label unreliability , Thus, label smoothing regularization is introduced .

Article method

Embedded layer

   Here the author adopts two ways to realize , One is static word embedding Glove, That is, already trained , The other is BERT Model , The author's construction methods are 2 Kind of , For the context “[CLS] + context + [SEP]”, For the target entity “[CLS] + target

• [SEP]”. In addition, the author also made an example that only BERT The comparison model of is called BERT-SPC, Its structure is “[CLS] + context + [SEP] + target + [SEP]”.

Attentional layer

   Here, the author draws lessons from the mechanism design of multi head attention 2 Ways of planting ：
Intra-MHA, Here the attention mechanism k and v All for context That is context , Is the internal attention mechanism , The formula is as follows ：
Inter-MHA Interactive attention mechanism ,q by context and k For the target entity , That is to learn the interaction between the target entity and the context , The formula is as follows ：

Point-wise Convolution Transformation

   stay MHA The author followed the output of a point by point convolution transform （PCT）, So as to further extract the information of attention mechanism , Point by point means that the kernel is 1 Of CNN, The formula is as follows ：

Target specific attention level

   In addition to passing the internal and interactive attention mechanisms respectively PCT outside , The author also applies another MHA To get the target specific context representation （ Tell the truth , I don't quite understand the meaning of Physics ） The formula is as follows ：

Output layer

   The final output is the splicing of the three, and then an average pooling , The formula is as follows ：

Label smoothing

   The idea of label smoothing is to make the final comparison result of the model not 0 or 1 This hard label , It is 0.1,0.9 Labels like this , So as to achieve the purpose of regularization ：

   Here, the smoothing score designed by the author is the reciprocal of the number of categories , The final loss is calculated as follows ：