Transformer -- Analysis and application of attention model

One 、 Summary

characteristic ：Transformer The attention model solves RNN Medium attention Low efficiency 、 The problem of deformation during training .
Purpose ： Machine translation

Two 、 Basic composition

encoder and decoder It is stackable .
The decoder gets the characteristics of the encoder , Combine the translated words to complete the translation .
As shown in the figure below , Red box encoder, The blue box indicates decoder,N=6.

Input ： Words to be translated (L A single hot code ) + Translated words (M A single hot code )
Output ： The probability of words
Embedded layer ： Through a transformation, the word's one-hot Represents mapping to a continuous space , Its dimension and model dimension 512 Agreement , You can use nn.Embedding Function implementation . Thus there are ： Sentences to be translated (L512) + Translated words (M512).
Location code ：i Indicates the dimension ,pos Indicates the position of the word

3、 ... and 、 Encoder

As shown above , The difference in RNN The attention mechanism in , Each word vector （512 dimension ） Through three linear transformations q、k、v.
Let's first decompose the multi head attention mechanism into a single one .
The calculation process is as follows ：
a11=q1k1,a12=q1k2,……,a1L=q1*kL
Re pass softmax Normalize to get a new a1j, Then for each pair a1j * vj Sum by accumulation , That is, the final output z.
Divided by the root sign dk（key Dimensions ） To solve the problem of large variance , take softmax Push to the problem of low gradient .
As shown below , It reflects the interdependence between features .

q、k、v The production process of is as follows ,512 Word vector of dimension and W matrix multiplication ：

Again , Output z It can also be realized by matrix operation , As shown below ：

thus , The multi head attention mechanism in the encoder gives h=8 Group W matrix , Will output 8 Group z Connect , Multiply by a weight matrix , The final output Z.
The formula in the paper is as follows ：


softmax Pre inclusion mask, Remove the end of the sentence padding Influence in the training process .

Encoding Add&Norm The operation is as follows , After the residual, use layernorm：

In the encoder FFN Layer 2 network ,512->2048->512, The formula is as follows ：

Four 、 decoder

The decoder also contains mask, in the light of decoder The input of , Not only to remove padding Influence , At the same time, in order to prevent decoder See the future information , Make an upper triangle of input mask, Maintain autoregressive properties .

5、 ... and 、 Self supervised learning

Here is a brief introduction cv Applications in the field .

6、 ... and 、Non-local

（ To be added ）

7、 ... and 、ViT

（ To be added ）

8、 ... and 、MAE

be based on ViT+BERT
Cover more picture blocks , Leave no redundancy
When encoding, only the uncovered
use Transformer decode