Maximum likelihood estimation, divergence, cross entropy

If this message , You can put things that were very uncertain before , To determine the , It shows that this information is very informative ！

This picture is to explain , How mathematicians define the process of information quantity ：

in other words , The amount of information about Argentina winning the championship = The amount of information about Argentina reaching the finals + The amount of information that Argentina won the finals , however x Itself is probability , So here 1/8 yes x1*x2 Result . And ：

This characteristic is log Peculiar . Then the base number is greater than 1 When ,log Monotone increasing , So add a minus sign to make it decrease separately .

The definition of this information quantity is completed .

Next, let's look at the definition of information entropy ：

 

The process of seeking information entropy is the process of seeking expectation ： Multiply the amount of information by the probability of the event , Then sum it . 

Information entropy , Describe the “ news ” The degree of uncertainty or confusion . And here it is “ news ” It can be regarded as a probability model . And find the information entropy of the probability model , Is to find the expectation of the probability model ！

If you want to compare the distribution differences between the two models , We need a concept called relative entropy （KL The divergence ） 

Here is a proof , Cross entropy must be greater than entropy ：

  

  So if you want the two models to be the closest , That is to find the minimum value of cross entropy . What deep learning does is to make the neural network model approach the human brain model . So the cross entropy of neural network and human brain model , It can be used as a loss function ！（ The smaller the cross entropy , The closer the two models are ）

Then the following function , It's actually , The expansion of cross entropy of a binary classification ：

  here Y It's human judgment , There are only two possibilities ,1 perhaps 0, and yhead, Is the output value of neural network , It's a probability . The cross entropy formed between them constitutes the loss function of two classifications .

---

 

Reference resources ：

《PyTorch Deep learning practice 》 Complete the collection _ Bili, Bili _bilibili

“ Cross entropy ” How to do the loss function ？ I understand “ The amount of information ”、“ The bit ”、“ entropy ”、“KL The divergence ”、“ Cross entropy ”_ Bili, Bili _bilibili