【pytorch】nn. Crossentropyloss() and nn NLLLoss()

One 、 Understanding of meaning The significance of cross entropy as a loss function

Cross entropy loss is often used for Multi classification function , Next, we understand the meaning of cross entropy as a loss function by disassembling the formula of cross entropy

Suppose we are making a n The problem of classification , The output of the model prediction is $[x_1,x_2,x_3,....,x_n]$
then , We need to define a loss function , Then the weight of the model is adjusted by back propagation , there We choose the loss function Cross entropy loss function ～

nn.CrossEntropyLoss() The formula of is ：
$$loss(x,class)=-log(\frac{e^{x_{[class]}}}{{\sum }_j{e}^{x_j}})=-x_{[class]}+log(\sum _j{e}^{x_j})$$

• x It's the prediction , It's a vector , The number of elements should be guaranteed by the model , Guarantee as many as the classification number
• class Represents the actual label of this sample , such as , The sample actually belongs to classification 2, that class=2
  $x_{[class]}$ Namely $x_2$, Is to take the second element in the test result vector , That is, take the predicted value corresponding to its real classification

It's over , Next , We need to disassemble the formula , Understand the formula
1、 First , The cross entropy loss function formula contains a basic part ：$softmax(x_i)=\frac{e^{x_i}}{{\sum }_j{e}^{x_j}}$
softmax The classification results are normalized ：$e^x$ First map the data to (0, 1] The range of , Then make the sum of all classification probabilities equal 1. after softmax After processing ,size It won't change , The meaning of each value is the probability that the sample is assigned to this classification .

2、 We want to make the prediction result , The probability of the value of the real classification is close to 100%. We take the value of the real classification ：
$\frac{e^{x_{[class]}}}{{\sum }_j{e}^{x_j}}$, We want its value to be 100%

3、 The meaning of being a loss function is ： When the prediction results are more Close to the real value , The closer the value of the loss function is to 0.
We put $\frac{e^{x_{[class]}}}{{\sum }_j{e}^{x_j}}$ take log, Take the opposite again , Can guarantee when $\frac{e^{x_{[class]}}}{{\sum }_j{e}^{x_j}}$ The more close to 100%, $loss=-log(\frac{e^{x_{[class]}}}{{\sum }_j{e}^{x_j}})$ The closer the 0.

Two 、 application ：

Suppose there is 4 A picture , Or say batch_ size=4. We need to put this 4 Pictures are classified into 5 Categories , for instance ： bird , Dog , cat , automobile , ship
After network calculation , We got the prediction results ：predict,size by [4, 5]
Its real label is label,size by [4]
Next use nn.CrossEntropyLoss() Calculation Predicted results predict and True value label The cross entropy loss of , Sure

import torch
import torch.nn as nn

# -----------------------------------------
#  Defining data : batch_size=4;  Altogether 5 A classification 
# label.size() : torch.Size([4])
# predict.size(): torch.Size([4, 5])
# -----------------------------------------
torch.manual_seed(100)
predict = torch.rand(4, 5)
label = torch.tensor([4, 3, 3, 2])
print(predict)
print(label)

# -----------------------------------------
#  Call function directly  nn.CrossEntropyLoss()  Calculation  Loss
# -----------------------------------------
criterion = nn.CrossEntropyLoss()
loss = criterion(predict, label)
print(loss)

---

nn.CrossEntropyLoss() It can be disassembled as follows 3 A step , Or it can be described as follows 3 Operation replacement , The result is the same ：

1. softmax： Classify the results of each picture softmax, softmax Detailed introduction
2. log： For the above results take log
   （ step 1 and step 2 Can be combined into nn.logSoftmax() ）
3. NLL：nn.NLLLoss(a, b) The operation of is from a Remove from b The corresponding value (b What's in store is index value ), Then remove the minus sign （ Take the opposite ）, Then sum and take the mean

import torch
import torch.nn as nn

torch.manual_seed(100)
predict = torch.rand(4, 5)
label = torch.tensor([4, 3, 3, 2])

softmax = nn.Softmax(dim=1)
nll = nn.NLLLoss()

temp1 = softmax(predict)
temp2 = torch.log(temp1)
output = nll(temp2, label)
print(output)   # tensor(1.5230)

Hand only version

import torch

torch.manual_seed(100)
predict = torch.rand(4, 5)
label = torch.tensor([4, 3, 3, 2])

# softmax
temp1 = torch.exp(predict) / torch.sum(torch.exp(predict), dim=1, keepdim=True)

# log
temp2 = torch.log(temp1)

# nll
temp3 = torch.gather(temp2, dim=1, index=label.view(-1, 1))
temp4 = -temp3
output = torch.mean(temp4)

print(output)    # tensor(1.5230)