Interpreting the knowledge in a neural network

problem ： Due to the complexity of the network structure , The cost of forecasting is too high , It is difficult to deploy the network to lightweight device users .

resolvent ： Use knowledge distillation to compress the model , Realize lightweight network .

Next, take this paper as the basis to understand knowledge distillation .

1、 Soft label and hard label

describe ： Hard label means that the correct value when we predict is 1, The wrong value is 0. Soft tags think that the wrong tags cannot be all zero , Because there is always a gap for wrong labels , This is shown below .

                      Hard tags         Soft label

        BMW            1               0.9

        Mercedes            0               0.6

      Garbage truck          0               0.3

          Carrot          0               0.001

It can be seen from the above description that , When we predict BMW , Except that the correct label is classified as 1, Other labels are 0, Also is 0 namely 1. But in fact, in the error category , BMW is more like Benz than carrot , This shows that there is also a gap in the information of the wrong label . So soft tags are introduced , Change the tag value to 0 To 1 Between the value of the , In this way, the information presented is richer .

2、 temperature coefficient T 

Soft tags can present the information of wrong tags , However, the gap between some wrong labels is not obvious , That is, the tag value is not soft enough . At this time, in order to make this inconspicuous gap more obvious , The author introduced the temperature coefficient T To change the original SoftMax function , namely ：

In style qi On behalf of the use of SoftMax Category probability of output 、zi For each category logit、T Is the temperature coefficient .

PS： If T Small , The information gap of error categories is small , But if T Too big , The label will be too soft , It is easy to lead to equalitarianism , It is difficult to achieve the predicted effect , Specifically T The selection effect of is as follows .

  In the figure , We can see that T The bigger it is , The smoother the curve . If T If it is too big, it is easy to appear equalitarianism , It will become unpredictable .

3、 Knowledge distillation network framework

  In the figure, the samples are fed into the teacher model and the student model for training , The teacher model is the original complex model , The student model is a simple model after compression . In calculating the loss , Distillation loss and student loss need to be calculated separately , The distillation loss is at a temperature of T Calculate the cross entropy of the output of the two networks of teachers and students , And the student loss is at the temperature of 1 Calculate the cross entropy of real tags and student network output , The formula is as follows .

Common in formula m A sample and n Categories ,y Represents the real label （ Hard tags ） ,P(xij) Indicates the soft label of students' network output ,Yij Indicates the soft label of teachers' network output ,λ Express 0 To 1 Weight coefficient between .

4、 Cross entropy gradient  

Suppose the teacher model is vi, The student model is zi, The soft target probability distribution of the two is qi and pi, The gradient of ：

 1、 Suppose the temperature coefficient can be infinite , The formula is transformed into ：

PS： use Taylor expansion of , If T infinity , Then the latter items can be ignored , At this point, let's take Taylor's first two , Just change to the above form .

2、 Suppose different samples logit The value is 0：

 PS： The sum of the two Sigma in the denominator is 0.

Therefore, in the special case where these two conditions are met , It turns into minimizing the mean square error .

5、 Conclusion  

In order to verify the effect of knowledge distillation , The author of the paper MINST A pre experiment was carried out on the handwritten numeral set , Then the performance of knowledge distillation is verified by the complex model distillation of speech recognition .