Deep learning ——（2） Several common loss functions

Great certainty , The smaller the entropy
Less certainty , The greater the entropy

1.L1 loss

advantage ： The first derivative is Changshu

shortcoming ： Non differentiable at zero

import torch
from torch import nn
criterion_l1=nn.L1Loss(reduction='mean')
input=torch.randn(3,5,requires_grad=True)
target=torch.randn(3,5)
loss_l1=criterion_l1(input,target)
# The calculation process 
loss_l11=(abs(target-input)).sum()/15
loss_l1,loss_l11

2.MSEloss

Second order function

shortcoming ： The gradient is very large , Will cause a gradient explosion ; After the derivation, the gradient will disappear

criterion_mse=nn.MSELoss()
loss_mse=criterion_mse(input,target)

# The calculation process 
loss_mse1=((target-input)**2).sum()/15
loss_mse,loss_mse1

3.smoothL1 loss

advantage ： Combined with the L1 and MSE The advantages of , Avoid zero point non derivative and gradient explosion

criterion_SML1=nn.SmoothL1Loss()
loss_sml1=criterion_SML1(input,target)
loss_sml1

4. Cross entropy loss

criterion_cross=nn.CrossEntropyLoss(reduction='mean',label_smoothing=0.0)
loss_cross=criterion_cross(input,target)
loss_cross

5.KL The divergence loss—— Relative entropy

criterion_KL=nn.KLDivLoss()
loss_KL=criterion_KL(input,target)
loss_KL