Why is it necessary to scale the attention before softmax (why divide by the square root of d_k)

from math import exp
from matplotlib import pyplot as plt
import numpy as np 
f = lambda x: exp(x * 2) / (exp(x) + exp(x) + exp(x * 2))
x = np.linspace(0, 100, 100)
y_3 = [f(x_i) for x_i in x]
plt.plot(x, y_3)
plt.show()

  The resulting graph is shown below ：

 

 

 

  be ：

1、self-attention Do you have to express it like this ？

Unwanted , Can depict Correlation , Similarity and other modeling methods are ok . Better be fast , Model is easy to learn , Expressive enough .

2、 There are other ways not to divide by the root dk Do you ？

Yes , ditto , As long as the gradient of each layer of parameters can be kept within the training sensitive range , Don't be too big , Don't be too small . Then this network is easier to train . There are ways , A better initialization method , Be similar to google Of T5 Model , Just do it during initialization .

Reference article ：

transformer Medium attention Why? scaled?

self-attention Why divide by the root d_k_tyler The blog of -CSDN Blog