[deep learning 05] cross entropy loss function

Basic loss function

The loss function acts

1. Calculate the difference between the actual output and the target
2. Provide the basis for us to update the output （ Back propagation ）

$$\begin{gathered} output:10,10,20 \\ target:30,20,50 \\ loss=|(30-10)|+|(20-10)|+\mid (50-10)\mid =70 \\ L1loss=70/3=23.33 \end{gathered}$$

Cross entropy

Two different models are compared , Entropy is needed as an intermediary . such as , The comparative value of gold and silver , They need to be converted into dollars , To compare

1. The amount of information

1. Different information , Contains different amounts of information , Suppose that the probability of winning the title against Argentina in the following table is 1/8,A My classmate told me that Argentina won the championship , Then the amount of information is very large （ Because it includes Argentina in the top four , finals ）;B My classmate told me that Argentina was in the finals , Then the amount of information is small .

   

2. hypothesis f(x):= The amount of information （:= Is a definition. ）,x It's information

   $f（orootTingSeizecrown）=f（orootTingInto\; the""）+f（orootTingwin了""）$

   Because the more uncertain the event is , The more information it contains , therefore The independent variable can be changed into the probability of the event

   Then there are ：

   $f（1/8）=f（1/4）+f(1/2)$

   meanwhile , Must also meet

   $P（orootTingSeizecrown）=P(orootTingInto\; the"")\ast P（orootTingwinhave\; to了""）$

   therefore

   $f(P(orootTingSeizecrown)\ast P(orootTingwinhave\; to了""))=f(P(orootTingInto\; the""))+f(P(orootTingwinhave\; to了""))$

   therefore , There must be... In the expression log

   And because the probability of an event is inversely proportional to the amount of information , So there is -log

2. entropy

entropy ： An event , From the original uncertainty to complete certainty , How big difficulty . And the expectation of information quantity , Is entropy $H(P)$

$$\begin{array}{c}H(P):=E\left(P_f\right)\\ ={\sum }_{i=1}^m{p}_i\cdot f\left(p_i\right)={\sum }_{i=1}^m{p}_i\left(-{\log }_2{p}_i\right)=-{\sum }_{i=1}^m{p}_i\cdot {\log }_2{p}_i\end{array}$$

$P_f$ Is the total amount of information ,$f(p_i)$ Is the amount of information about the event ,$p_i$ Is the probability that the event will occur

The smaller the cross entropy , The closer the two models are

3. Relative entropy （KL The divergence ）

$f_Q(q_i)$ Express Q The amount of information in the system ;$f_P(p_i)$ yes P The amount of information in the system

$D_{K}L(P||Q)$ Represents the relative entropy of two systems , Or say KL The divergence

$D_{K}L(P||Q)$ With P Benchmarking , Think about it P、Q What's the difference

$D_{K}L(Q||P)$ Said to Q Benchmarking
$f_Q\left(q_i\right)-f_P\left(p_i\right)$ Indicates an event , stay Q The amount of information in the system , subtract P The amount of information in the system

$$\begin{array}{l}{\mathbf{D}}_{\mathbf{K}\mathbf{L}}(\mathbf{P}\Vert \mathbf{Q})\\ :={\sum }_{i=1}^m{p}_i\cdot \left(f_Q\left(q_i\right)-f_P\left(p_i\right)\right)\\ ={\sum }_{i=1}^m{p}_i\cdot \left(\left(-{\log }_2{q}_i\right)-\left(-{\log }_2{p}_i\right)\right)\\ ={\sum }_{i=1}^m{p}_i\cdot \left(-{\log }_2{q}_i\right)-{\sum }_{mi=1}^m{p}_i\cdot \left(-{\log }_2{p}_i\right)\end{array}$$

${\sum }_{mi=1}^m{p}_i\cdot (-{\log }_2{p}_i)$：P The entropy of , Because we put P Customized benchmark , So when looking at divergence , Just think about it ${\sum }_{i=1}^m{p}_i\cdot \left(-{\log }_2{q}_i\right)$, This part , Namely Cross entropy 了

1. Dichotomous problem

   Cross entropy should include all possible results , The result of the second classification is ： yes / no , So there must be $(1-x_i)\cdot {\log }_2\left(1-y_i\right)$
   $$\begin{array}{ll}\mathbf{H}(\mathbf{P},\mathbf{Q})& =-{\sum }_{i=1}^n\left(x_i\cdot {\log }_2{y}_i+\left(1-x_i\right)\cdot {\log }_2\left(1-y_i\right)\right)\end{array}$$

2. Multiple classification problem

$$\mathbf{H}(\mathbf{P},\mathbf{Q})=\sum _{i=1}^m{p}_i\cdot \left(-{\log }_2{q}_i\right)$$

Be careful ：python in log What's the bottom line e, namely ln

4. Cross entropy

pytorch The cross entropy in is a little different , It is a softmax Function as the probability of the event

$w_c$ Weight.

Theory is hard , It's really easy to use , Just a code thing ~

loss_fn = nn.CrossEntropyLoss() #  Cross entropy loss