Understanding of entropy and cross entropy loss function

Several well explained links
Detailed explanation of entropy and cross entropy loss function
Detailed explanation and formula derivation of information entropy

entropy

Let's first understand what entropy is ？
The concept of entropy in information theory was put forward by Shannon for the first time , The aim is to find an efficient / A method of encoding information losslessly ： Efficiency is measured by the average length of encoded data , The smaller the average length, the more efficient ; At the same time, it also needs to meet “ Nondestructive ” Conditions , That is, the original information cannot be lost after coding . such , Shannon put forward the definition of entropy ： Minimum average coding length of lossless coding event information .

Information entropy

The amount of information ： Measurement of information , It is used to measure the amount of information , Its value is related to the specific event .

The size of information is related to the probability of random events .
The less probability things happen, the more information they generate , Such as the earthquake in Hunan ;
The greater the probability of something happening, the smaller the amount of information , If the sun rises from the East .

Derive information entropy
Information quantity measures the information brought about by a specific event , Entropy, on the other hand, is the expectation of the amount of information that may be generated before the results are obtained —— Consider all possible values of the random variable , That is, the expectation of the amount of information brought by all possible events .

The formula of information entropy
Information entropy can be understood as seeking ${\log }_{2}p$ The expectation of , namely ：
$$H(X)={\mathbb{E}}_{x\backsim P}[-\log x]=-\sum _{i=1}^{n}p(x_i)\log p(x_i)$$
The above formula is the code length calculation formula , It is also information entropy .
for instance ：

There are four kinds of weather , Corresponding to four codes , Calculate the entropy directly ：
entropy = 1 * 50% + 2 * 25% + 3 * 12.5% + 3 * 12.5% = 1.75

Relative entropy （KL The divergence ）

KL The divergence （Kullback-Leibler divergence） It's also called relative entropy .
If random variable X There are two separate probability distributions p(x) and p(x), The difference between these distributions can be measured by relative entropy ,
Relative entropy is defined as follows ：
$$D_{KL}(P||Q)=\sum _{i=1}^{n}P(x_i)\log \frac{P(x_i)}{Q(x_i)}$$
1、 The relative entropy is nonnegative , Its size can be used to measure the difference between two distributions , If and only if P and Q With exactly the same distribution , The value of relative entropy is 0.
2、 Relative entropy is not symmetric , For certain P and Q,
$D_{KL}(P||Q)\ne D_{KL}(Q||P)$

Cross entropy

Cross entropy is used $H(P,Q)$ Express , Implied use P Calculate expectations , Use Q Calculate the length of the code
Calculation formula of cross entropy ：
$$H(P,Q)={\mathbb{E}}_{x\backsim P}[-\log Q(x)]$$

Cross entropy as a loss function

for instance , Suppose an animal photo data set has 5 Animals , And there is only one animal in each picture , The label of each photo is one-hot code .

The probability of the first picture being a dog is 100%, The probability of other animals is 0, The rest of the pictures are the same .
Suppose two machine learning models predict the first picture ：Q1 and Q2, But a photo is really labeled [1,0,0,0,0].

Calculate the cross entropy respectively ：

The smaller the cross entropy is , The more accurate the prediction is , It can be seen that the model Q2 Better prediction .