Loss function -- cross entropy loss function

One 、 Definition

All algorithms in machine learning need to maximize or minimize a function , This function is called “ Objective function ”. among , We usually minimize a class of functions , be called “ Loss function ”. It can be based on the predicted results , Measure the prediction ability of the model .

Loss functions can be roughly divided into two categories ： Loss function of classification problem and Loss function of regression problem

Two 、 Cross entropy loss function ：

Knowledge preparation ：

entropy ： Indicates the degree of uncertainty of a system , Or a systematic The level of confusion

1、 Information entropy ： Introduce entropy into information theory , Name it “ Information entropy ”

Formula application ：
The information entropy here means The level of confusion Or is it uncertainty

2、 KL The divergence （ Relative entropy ）：

KL The divergence ： Is an asymmetric measure of the difference between two probability distributions .
Popular parlance is used to measure the distance between two different distributions of the same random variable

Formula application ：

The formula is distorted ：

Cross entropy ：

Cross entropy is mainly used to measure the same random variable X Of Forecast distribution Q And the real distribution P Between disparity

The cross entropy obtained here means the difference between the real label and the real label , The smaller the better.

Conclusion ：

It can be seen from the picture that , If it's not the real label , Its label P(X)=0 So the result of calculation has nothing to do with it

1、 The more accurate the prediction , The smaller the cross entropy
2、 The cross entropy is only related to the predicted probability value of the real label
So we can simplify the cross entropy formula ：

---

Two classification cross entropy formula ：

---

Why use cross entropy instead of KL The divergence ？

---

Softmax The formula

1、 Convert numbers into probabilities
2、 A sharp tool for data normalization

---

Sigmoid

1、Sigmoid The function is also called Logistic function
2、 The value range is （0,1）
3、 Common functions of Neural Networks
4、 Often used as a secondary classification

---

Common cross entropy loss function types

Cross entropy loss function ：

The specific process ：