[pytorch] softmax function

Students who have done many classification tasks must know softmax function .softmax function , Also known as normalized exponential function . It's a binary function sigmoid Generalization on multi classification , The purpose is to show the results of multi classification in the form of probability . The following figure shows softmax The calculation method of ：

Now let's explain why softmax It's in this form .

First , We know that probability has two properties ：1） The probability of prediction is non negative ;2） The sum of the probabilities of various prediction results is equal to 1.

softmax It is to convert the prediction result from negative infinity to positive infinity into probability according to these two steps .

1） Convert the forecast results to non negative numbers

The following figure for y=exp(x） Image , We can know that the value range of the exponential function is from zero to positive infinity .softmax The first step is to transform the prediction results of the model into an exponential function , This ensures that the probability is nonnegative .

2） The sum of the probabilities of various prediction results is equal to 1

To ensure that the sum of the probabilities of each prediction result is equal to 1. We only need to normalize the converted results . The way is Divide the converted results by the sum of all converted results , It can be understood as the percentage of the converted results in the total . So we get the approximate probability .

Here is an example , Suppose the prediction result of the model for a three classification problem is -3、1.5、2.7. We need to use it. softmax Convert model results into probability . Steps are as follows ：

1） Convert the forecast results to non negative numbers

y1 = exp(x1) = exp(-3) = 0.05

y2 = exp(x2) = exp(1.5) = 4.48

y3 = exp(x3) = exp(2.7) = 14.88

2） The sum of the probabilities of various prediction results is equal to 1

z1 = y1/(y1+y2+y3) = 0.05/(0.05+4.48+14.88) = 0.0026

z2 = y2/(y1+y2+y3) = 4.48/(0.05+4.48+14.88) = 0.2308

z3 = y3/(y1+y2+y3) = 14.88/(0.05+4.48+14.88) = 0.7666

To sum up softmax How to convert multi classification output into probability , It can be divided into two steps ：

1） molecular ： By exponential function , Map real output to zero to positive infinity .

2） The denominator ： Add all the results , Normalize .

The following picture shows Stanford University CS224n Most softmax The explanation of ：