[paper reading] mean teachers are better role models

download

1. Abstract

Recently proposed Temporal Ensembling The most advanced results have been achieved in several semi supervised learning benchmarks . It maintains tag predictions for each training example EMA, And punish predictions that are inconsistent with this goal . However , Because every goal epoch Only change once , So when learning large data sets ,Temporal Ensembling Become clumsy . To overcome this problem , We proposed Mean Teacher, This is a kind of Average model weight Rather than label prediction . As an additional benefit ,Mean Teacher Improve the accuracy of the test , And you can use ratio Temporal Ensembling Less tags for training .

2. Pre knowledge

While reading the abstract of the paper , At the same time, it is also accompanied by ignorance , The reason is simply the lack of pre knowledge in this field . This section mainly introduces the pre knowledge I supplemented in the process of understanding the paper .

2.1. Temporal Ensembling

We can translate it into Time integration , The first sentence in the abstract is a tribute to the discovery of our predecessors , So this is state of art Proposed . Want to know the knowledge points in this paper , You can finish reading the previous Blog .

2.2. EMA

EMA(exponential moving average), Also called exponential moving average , It is a type of average commonly used in time series analysis . Simply speaking ,EMA Is a weighted average . among , An important feature of it is that with the passage of time , Old observations will show exponential decay . equation 1, It means EMA The recurrence formula of , Details can be found in Blog .

$$S_t=\{\begin{array}{ll}{S}_0,& t=1\\ (1-\alpha )S_{t-1}+\alpha X_t,& t\ge 2\end{array}\text{\hspace{1em}(1)}$$

3. Algorithm description

The algorithm innovation proposed in this paper is based on the previous article The paper Of , The main change is the cost of consistency , This can be understood as unsupervised loss . The algorithm in this paper is essentially to maintain two models ,Teacher and Student, The result of two norm operation with the same input through the output of two models is regarded as unsupervised loss .$\Pi$ model In essence, only one model is maintained , Just there will be Dropout.$TemporalEnsembling$ Only one model is maintained , Just put the model in each epoch The output of EMA. This article is more direct , Maintain two models directly , Make the parameters of the two models EMA, Macroscopically, it can be seen as a model that imparts its own experience to another model , It is described in the paper as Mean Teacher .