[loss function] entropy / relative entropy / cross entropy

Easy to understand but not precise ：
entropy ： All the information about a possibility
Relative entropy （KL The divergence ）： The difference in the amount of information between the real distribution and the predicted distribution （ real - forecast ）, The value of the novel is close to the truth .
Cross entropy ： from kl Derived from the deformation of divergence formula , The first half of the formula is entropy, and the second half is cross entropy . Because the entropy in front is constant , Therefore, it is more convenient to calculate the loss directly with cross entropy when optimizing .

Blog records are convenient for review and sorting , If there is any mistake, please advise , thank you ！

reference

https://blog.csdn.net/tsyccnh/article/details/79163834