[paper reading] temporary binding for semi-superior learning

download
Code address

1. Abstract

In this paper , We propose a simple and effective method , For training deep neural networks in a semi supervised environment , Only a small part of the training data is marked . We introduced Self integration , We use the training output of the network in different periods to form a consensus prediction of unknown tags , most important of all , Under different regularization and input enhancement conditions . Compared with the network output in the recent training period , This integrated prediction can be expected to be a better predictor of unknown tags , Therefore, it can be used as the goal of training .

Notice: The article states straight to the point that this is a method in semi supervised deep neural network . The main contribution is to propose for Model disturbance The idea of , Two models are proposed ,$\mathbf{\Pi }$ model, Temporal ensembling.

2. Algorithm description

2.1. $\mathbf{\Pi }$ model

Through the flow chart and pseudo code in the paper , We can clearly understand the general flow of the algorithm . Some of the small details , It may need to be found when it reappears , The words here , Just record your questions , If you look back later, read carefully to answer .
Q1： The depth dependence of the proposed model is expressed in the paper Input Augment and Dropout, stay $\Pi$ Model Between perturbed model and undisturbed model Input Augment Is it consistent .
Q2: Why is there a difference between the parameters of supervised loss and unsupervised loss in pseudo code loss $C$, among $C$ Indicates the number of data labels .

2.2. Temporal ensembling

Journal entry ： First of all, the description of the paper is very clear , You can clearly understand the general flow of the algorithm only by looking at the pseudo code . The second is with $\Pi$ model comparison ,Temporal ensembling The unsupervised loss of is based on the previous model epoch The error between the output of and the current output . It is pointed out in the article that ,Temporal ensembling than $\Pi$ model faster , as a result of Temporal ensembling Every batch Just do a forward operation , and $\Pi$ model There are two forward operations . In fact, the essence of faster speed here is Space for time , Similar to caching .

TODO: There are some places in the paper trick The author did not explain , We should acquiesce to the knowledge that everyone knows , But I don't know , You can get to know . for instance ：
$$Z\leftarrow \alpha Z+(1-\alpha )z\text{\hspace{1em}(1)}$$
$$\tilde{z}\leftarrow Z/(1-{\alpha }^t)\text{\hspace{1em}(2)}$$.