Position encoding practice in transformer

In recent years ,transformer Because it can realize parallel computing and solve the dependency problem of long sequences, it is in nlp Areas and cv The field is brilliant .
The schematic diagram is as follows ：

Here we mainly focus on a small part , namely position encoding part , because transformer Eliminate circular dependency , In order to reflect the location attribute , So encode the position of each element .
The code is as follows , As for why it is written like this , You can refer to the author's original , Or refer to the article .https://zhuanlan.zhihu.com/p/338592312
The code is as follows ：

class PositionalEncoding(torch.nn.Module):
    def __init__(self, d_model, max_len=5000):
        super(PositionalEncoding, self).__init__()

        pe = torch.zeros(max_len, d_model)
        position = torch.arange(0, max_len, dtype=torch.float).unsqueeze(1)
        
        div_term = torch.exp(torch.arange(0, d_model, 2).float() * (-math.log(10000.0) / d_model))
        
        pe[:, 0::2] = torch.sin(position * div_term)
        pe[:, 1::2] = torch.cos(position * div_term)
        
        pe = pe.unsqueeze(0).transpose(0, 1)#(max-len,1,d_model)
        
        self.register_buffer('pe', pe)

    def forward(self, x):
        x = x + self.pe[:x.size(1), :].squeeze(1)
        #x = x + self.pe[:x.size(1), :]
        return x

In order to test , We define two input matrices , Full respectively 0、 whole 1tensor.

d_model = 4   
a=torch.zeros(2,3,4)
pos=PositionalEncoding(d_model)
b=pos(a)
c=torch.ones(2,3,4)
b1=pos(c)

Obviously , The input matrix is

Output is b,b1, As shown below ：;

It can be seen that , Are based on input , Add a fixed value , And those fixed values are encoded , It's not about input , And d_model of ,d_model It can be understood as a word embedding size .