Label smoothing

Catalog

1. What problems does label smoothing mainly solve ？

2. How does label smoothing work ？

3. Label smoothing formula

4. Code implementation

---

Label smoothing （label smoothing） come from GoogleNet v3

About one-hot For detailed knowledge of coding, see ：One-hot code

1. What problems does label smoothing mainly solve ？

Conventional one-hot code It will bring problem ： The generalization ability of the model cannot be guaranteed , Making the network overconfident can lead to over fitting .
Total probability and 0 probability Encourage between the category and other categories The gap should be widened as much as possible , And from the boundedness of the gradient , It's hard to adapt. Meeting Cause the model to believe too much in the category of prediction . and Label smoothing Can alleviate this problem .
 

2. How does label smoothing work ？

Label smoothing is hold one-hot Medium probability is 1 The item of is attenuated , Avoid overconfidence , The decaying part of confidence Be divided equally into each category .

for example ：

One 4 Classification task ,label = （0,1,0,0）

labeling smoothing = （,1-0.001+,,）=（0.00025,0.99925,0.00025,0.00025）

  among , The probability adds up to 1.

 

3. Label smoothing formula

Cross entropy （Cross Entropy）：

among ,q For tag value ,p To predict the result ,k For the category . namely q by one-hot Coding results .

labeling smothing： take q Smooth the label to q', Let the model output p Distribution to approximate q'.

, among u(k) Is a probability distribution , Here the Uniform distribution （）, Then we get

         among , Is the original distribution q, ϵ ∈(0,1) It's a super parameter .

         It can be seen from the above formula that , In this way label Yes ϵ   Probability comes from uniform distribution , 1−ϵ The probability comes from the original distribution . This is equivalent to In the original label Add noise to the , Let the predicted value of the model Don't focus too much on categories with high probability , Put some probabilities in the lower probability category .

Therefore, the cross entropy loss function after label smoothing is ：

How did you get this formula ？

take q'(k|x) Bring in the cross entropy loss function ：

In this way, we get the label smoothing formula .

4. Code implementation

class LabelSmoothingCrossEntropy(nn.Module):
    def __init__(self, eps=0.1, reduction='mean', ignore_index=-100):
        super(LabelSmoothingCrossEntropy, self).__init__()
        self.eps = eps
        self.reduction = reduction
        self.ignore_index = ignore_index

    def forward(self, output, target):
        c = output.size()[-1]
        log_pred = torch.log_softmax(output, dim=-1)
        if self.reduction == 'sum':
            loss = -log_pred.sum()
        else:
            loss = -log_pred.sum(dim=-1)
            if self.reduction == 'mean':
                loss = loss.mean()


        return loss * self.eps / c + (1 - self.eps) * torch.nn.functional.nll_loss(log_pred, target,
                                                                                   reduction=self.reduction,
                                                                                   ignore_index=self.ignore_index)