Self attention and multi head attention

We met before A general attention model . This article will introduce self attention and multi attention , For the follow-up introduction Transformer Do matting .

Self attention

If attention in the attention model is calculated completely based on eigenvectors , Then call this kind of attention self attention ：

Picture changed from ：[1]

for example , We can use the weight matrix ${\mathbf{W}}_K\in {\mathbb{R}}^{d_k\times d_f}$、${\mathbf{W}}_V\in {\mathbb{R}}^{d_v\times d_f}$ and ${\mathbf{W}}_Q\in {\mathbb{R}}^{d_q\times d_f}$ For the characteristic matrix $\mathbf{F}=[{\mathbf{f}}_1,\dots ,{\mathbf{f}}_{n_f}]\in {\mathbb{R}}^{d_f\times n_f}$ Make a linear transformation , obtain

key (Key) matrix
$$\begin{array}{rl}\mathbf{K}& ={\mathbf{W}}_K\mathbf{F}\\ & ={\mathbf{W}}_K[{\mathbf{f}}_1,\dots ,{\mathbf{f}}_{n_f}]\\ & =\left[{\mathbf{k}}_1,\dots ,{\mathbf{k}}_{n_f}\right]\in {\mathbb{R}}^{d_k\times n_f}\end{array}$$

value (Value) matrix
$$\begin{array}{rl}\mathbf{V}& ={\mathbf{W}}_V\mathbf{F}\\ & ={\mathbf{W}}_V[{\mathbf{f}}_1,\dots ,{\mathbf{f}}_{n_f}]\\ & =\left[{\mathbf{v}}_1,\dots ,{\mathbf{v}}_{n_f}\right]\in {\mathbb{R}}^{d_v\times n_f}\end{array}$$

Inquire about (Query) matrix
$$\begin{array}{rl}\mathbf{Q}& ={\mathbf{W}}_Q\mathbf{F}\\ & ={\mathbf{W}}_Q[{\mathbf{f}}_1,\dots ,{\mathbf{f}}_{n_f}]\\ & =\left[{\mathbf{q}}_1,\dots ,{\mathbf{q}}_{n_f}\right]\in {\mathbb{R}}^{d_q\times n_f}\end{array}$$

$\mathbf{Q}$ Each column of ${\mathbf{q}}_i$ Are used as queries for the attention model . When using query vectors ${\mathbf{q}}_i$ When calculating attention , Generated context vector ${\mathbf{c}}_i$ Will summarize the eigenvectors in the query ${\mathbf{q}}_i$ For important information .

First , Query pair ${\mathbf{q}}_i,i=1,2,...,n_f$ Calculate key vector ${\mathbf{k}}_j$ The attention score of ：

$$\underset{1\times 1}{e_{i,j}}=\mathrm{score}\left(\underset{d_q\times 1}{{\mathbf{q}}_i},\underset{d_k\times 1}{{\mathbf{k}}_j}\right),j=1,2,...,n_f$$

Inquire about ${\mathbf{q}}_i$ Indicates a request for information . Attention score $e_{i,j}$ Indicates that according to the query ${\mathbf{q}}_i$, Bond vector ${\mathbf{k}}_j$ How important is the information contained in . Calculate the score of each value vector , Get information about the query ${\mathbf{q}}_i$ Attention score vector ：

$${\mathbf{e}}_{\mathbf{i}}=[e_{i1},e_{i2},...,e_{i,n_f}{]}^T$$

then , Use the alignment function $\mathrm{align}()$ Align ：
$$\underset{1\times 1}{a_{i,j}}=\mathrm{align}\left(\underset{1\times 1}{e_{i,j};}\underset{n_f\times 1}{{\mathbf{e}}_{\mathbf{i}}}\right),j=1,2,...,n_f$$

Get the attention weight vector ：${\mathbf{a}}_i=[a_{i,1},a_{i,2},...,a_{i,n_f}{]}^T$.

Finally, the context vector is calculated ：
$$\underset{d_v\times 1}{{\mathbf{c}}_i}=\sum _{j=1}^{n_f}\underset{1\times 1}{a_{i,j}}\times \underset{d_v\times 1}{{\mathbf{v}}_j}$$

Summarize the above steps , The self attention calculation expression is ：

$${\mathbf{c}}_i=\text{ self-att }({\mathbf{q}}_i,\mathbf{K},\mathbf{V})\in {\mathbb{R}}^{d_v}$$
because ${\mathbf{q}}_i={\mathbf{W}}_Q{\mathbf{f}}_i$, So we can say that the context vector ${\mathbf{c}}_i$ Include all eigenvectors （ Include ${\mathbf{f}}_i$） For a particular eigenvector ${\mathbf{f}}_i$ For important information . for example , For language , This means that self attention can extract the relationship between word features （ Verbs and nouns ; Pronouns and nouns, etc ）, If ${\mathbf{f}}_i$ Is the eigenvector of a word , Then self attention can be calculated from other words ${\mathbf{f}}_i$ For important information . For the image , Self attention can get the relationship between the features of each image region .

Calculation $\mathbf{Q}$ Context vectors for all query vectors in , Get the output from the attention layer ：
$$\mathbf{C}=\text{ self-att }(\mathbf{Q},\mathbf{K},\mathbf{V})=[{\mathbf{c}}_1,{\mathbf{c}}_2,...,{\mathbf{c}}_{n_f}]\in {\mathbb{R}}^{d_v\times n_f}$$

Long attention

Multi head attention works by using multiple different versions of the same query to implement multiple attention modules in parallel . The idea is to use different weight matrices to query $\mathbf{q}$ Get multiple queries by linear transformation . Each newly formed query essentially requires different types of relevant information , This allows the attention model to introduce more information into the context vector computation .

Bulls pay attention to $d$ Each header has its own multiple query vectors 、 Key matrix and value matrix ：${\mathbf{q}}^{(l)},{\mathbf{K}}^{(l)}$ and ${\mathbf{V}}^{(l)}$,$l=1,\dots ,d$.

Inquire about ${\mathbf{q}}^{(l)}$ By the original query $\mathbf{q}$ After linear transformation, we get , and ${\mathbf{K}}^{(l)}$ and ${\mathbf{V}}^{(l)}$ It is $\mathbf{F}$ After linear transformation, we get . Each attention head has its own learnable weight matrix ${\mathbf{W}}_q^{(l)}、{\mathbf{W}}_K^{(l)}and{\mathbf{W}}_V^{(l)}$. The first $l$ Head of the query 、 The keys and values are calculated as follows ：

$$\underset{d_q\times 1}{{\mathbf{q}}^{(l)}}=\underset{d_q\times d_q}{{\mathbf{W}}_q^{(l)}}\times \underset{d_q\times 1}{q},$$

$$\underset{d_k\times n_f}{{\mathbf{K}}^{(l)}}=\underset{d_k\times d_f}{{\mathbf{W}}_K^{(l)}}\times \underset{d_f\times n_f}{F}$$

$$\underset{d_v\times n_f}{{\mathbf{V}}^{(l)}}=\underset{d_v\times d_f}{{\mathbf{W}}_V^{(l)}}\times \underset{d_f\times n_f}{F}$$

Each header creates its own pair of queries $\mathbf{q}$ And input matrix $\mathbf{F}$ It means , This allows the model to learn more information . for example , When training language models , An attentional head can learn to pay attention to certain verbs （ For example, walking 、 Drive 、 Buy ） And noun （ for example , Student 、 automobile 、 Apple ） The relationship between , The other attentional head learns to focus on pronouns （ for example , He 、 she 、it） The relationship with nouns .

Each head will also create its own attention score vector ${\mathbf{e}}_i^{(l)}={\left[e_{i,1}^{(l)},\dots ,e_{i,n_f}^{(l)}\right]}^T\in {\mathbb{R}}^{n_f}$, And the corresponding attention weight vector ${\mathbf{a}}_i^{(l)}={\left[a_{i,1}^{(l)},\dots ,a_{i,n_f}^{(l)}\right]}^T\in {\mathbb{R}}^{n_f}$

then , Each header generates its own context vector ${\mathbf{c}}_i^{(l)}\in {\mathbb{R}}^{d_v}$, As shown below ：

$$\underset{d_v\times 1}{{\mathbf{c}}_i^{(l)}}=\sum _{j=1}^{n_f}\underset{1\times 1}{a_{i,j}^{(l)}}\times \underset{d_v\times 1}{{\mathbf{v}}_j^{(l)}}$$

Our goal is still to create a context vector as the output of the attention model . therefore , The context vectors generated by each attention head are connected into a vector . then , Use the weight matrix ${\mathbf{W}}_O\in {\mathbb{R}}^{d_c\times d_{v}d}$ Perform a linear transformation on it ：
$$\underset{d_c\times 1}{{\mathbf{c}}_i}=\underset{d_c\times d_{v}d}{{\mathbf{W}}_O}\times \mathrm{concat}\left(\underset{d_v\times 1}{{\mathbf{c}}_i^{(1)}};\dots ;\underset{d_v\times 1}{{\mathbf{c}}_i^{(d)}}\right)$$

This ensures that the final context vector ${\mathbf{c}}_i\in {\mathbb{R}}^{d_c}$ Conform to the target dimension

Reference resources ：

[1] A General Survey on Attention Mechanisms in Deep Learning https://arxiv.org/pdf/2203.14263v1.pdf