Deep understanding of softmax

​

Preface

This code is based on Pytorch Realization .

One 、softmax Definition and code implementation of

1.1 Definition

$$softmax(x_i)=\frac{exp(x_i)}{{\sum }_j^{n}exp(x_j)}$$

1.2 Code implementation

def softmax(X):
    '''  Realization softmax  Input X The shape of is [ Number of samples , Output vector dimension ] '''
    return torch.exp(X) / torch.sum(torch.exp(X), dim=1).reshape(-1, 1)

>>> X = torch.randn(5, 5)
>>> y = softmax(X)
>>> torch.sum(y, dim=1)
tensor([1.0000, 1.0000, 1.0000, 1.0000])

Two 、softmax The role of

softmax The output of the linear layer can be normalized and calibrated ： Ensure that the output is nonnegative and the sum is 1.
Because if you directly regard the non normalized output as probability , Will exist 2 Some questions ：

1. The output of the linear layer does not limit the sum of the output numbers of each neuron to 1;
2. Depending on the input , The output of the linear layer may be negative .
   this 2 Point violates the basic axiom of probability .

3、 ... and 、softmax Overflow on (overflow) Overflow with bottom (underflow)

3.1 It spills over

When $x_i$ When the value of is too large , The value of index operation is too large , If it is beyond the accuracy range , Then overflow .

>>> torch.exp(torch.tensor([1000]))
tensor([inf])

3.2 underflow

When the vector $\mathbf{x}$ Each element of $x_i$ When the values of are negative numbers with large absolute values , be $exp(x_i)$ The value of is very small, and it is taken down beyond the accuracy range 0, The denominator ${\sum }_{j}exp(j)$ The values for 0.

>>> X = torch.ones(1, 3) * (-1000)
>>> softmax(X)
tensor([[nan, nan, nan]])

3.3 Avoid spillovers

Reference resources ¹ The technique in ：

1. Find vector $\mathbf{x}$ Maximum of :
   $$c=max(\mathbf{x})$$
2. $softmax$ The molecules of 、 Divide the denominator by $c$
   $$softmax(x_i-c)=\frac{exp(x_i-c)}{{\sum }_j^{n}exp(x_j-c)}=\frac{exp(x_i)exp(-c)}{{\sum }_j^{n}exp(x_i)exp(-c)}=softmax(x_i)$$
   After the above transformation , The maximum value of the molecule becomes $exp(0)=1$, Avoid upper overflow ;
   At least $+1$, Avoid denominator 0 Cause lower overflow .
   $$\begin{gathered} \sum _j^{n}exp(x_j-c)=exp(x_i-c)+exp(x_2-c)+...+exp(x_{max}-c) \\ =exp(x_1-c)+exp(x_2-c)+...+1 \end{gathered}$$

def softmax_trick(X):
    c, _ = torch.max(X, dim=1, keepdim=True)
    return torch.exp(X - c) / torch.sum(torch.exp(X - c), dim=1).reshape(-1, 1)
>>> X = torch.tensor([[-1000, 1000, -1000]])
>>> softmax_trick(X)
tensor([0., 1., 0.])
>>> softmax(X)
tensor([[0., nan, 0.]])

pytorch The implementation of has been done to prevent overflow , therefore , Its operation results are similar to softmax_trick Agreement .

import pytorch.nn.functional as F
>>> X = torch.tensor([[-1000., 1000., -1000.]])
>>> F.softmax(X, dim=1)
tensor([[0., 1., 0.]])

3.4 Log-Sum_Exp Trick²（ take log operation ）

1. Avoid spillage
Logarithmic operation can change multiplication into addition , namely ：$log(x_1{x}_2)=log(x_1)+log(x_2)$. When two very small numbers $x_1、x_2$ Multiplying time , The product becomes smaller , If the accuracy is exceeded, it will overflow ; The logarithmic operation turns the product into addition , Reduce the risk of lower overflow .
2. Avoid overflow
$log-softmax$ The definition of ：
$$\begin{array}{rl}log-softmax& =log[softmax(x_i)]\\ & =log(\frac{exp(x_i)}{{\sum }_j^{n}exp(x_j)})\\ & =x_i-log[\sum _j^{n}exp(x_j)]\end{array}$$
Make $y=log{\sum }_j^{n}exp(x_j)$, When $x_j$ When the value of is too large ,$y$ There is a risk of spillover , therefore , with 3.3 The same in Trick:
$$\begin{array}{rl}y& =log\sum _j^{n}exp(x_j)\\ & =log\sum _j^{n}exp(x_j-c)exp(c)\\ & =c+log\sum _j^{n}exp(x_j-c)\end{array}$$
When $c=max(\mathbf{x})$ when , Avoid overflow .
here ,$log-softmax$ The calculation formula of becomes ：（ In fact, it is equivalent to directly 3.3 Chaste Trick Take the logarithm ）
$$log-softmax=(x_i-c)-log\sum _j^{n}exp(x_j-c)$$
Code implementation ：

def log_softmax(X):
	c, _ = torch.max(X, dim=1, keepdim=True)
	return X - c - torch.log(torch.sum(torch.exp(X-c), dim=1, keepdim=True))
>>> X = torch.tensor([[-1000., 1000., -1000.]])
>>> torch.exp(log_softmax(X))
tensor([[0., 1., 0.]])
# pytorch API Realization 
>>> torch.exp(F.log_softmax(X, dim=1))
tensor([[0., 1., 0.]])

3.5 log-softmax And softmax The difference between ³

combination 3.3 Chaste Trick And my own understanding ：

1. stay pytorch In the implementation of ,softmax The result of the operation is equivalent to log_softmax The result of is exponentially calculated

>>> X = torch.tensor([[-1000., 1000., -1000.]])
>>> torch.exp(F.log_softmax(X, dim=1)) == F.softmax(X)
tensor([[True, True, True]])

2. Use $log$ It is more convenient to derive after operation , It can speed up the speed of back propagation ⁴
   $$\begin{array}{rl}\frac{\partial }{\partial x_i}logsoftmax& =\frac{\partial }{\partial x_i}[x_i-log\sum _j^{n}exp(x_j)]\\ & =1-softmax(x_i)\end{array}$$

​

---

1. BTTB You don't know softmax︎

2. The Log-Sum_Exp Trick︎

3. log-softmax And softmax The difference between ︎

4. Hands-on deep learning ：softmax Return to ︎