Paddlepaddle 20 implementation and use of exponentialmovingaverage (EMA) (support static graph and dynamic graph)

Exponentially moving average （ExponentialMovingAverage,EMA） It's a exponentially decreasing weighted moving average , Each update is based on the weight reserved last time decay To attenuate . The calculation method is pram_n'=(pram_n'-1)*decay+(1-decay)*pram_n, among pram_n' by EMA Saved in n Step weight ,pram_n Is the weight calculated normally by the algorithm .
EMA It's an iterative operation , In the n Step by step , The first k The update parameters are attenuated decay^(n-k) times .EMA The essence is a strategy of learning rate attenuation , See blog for details （https://www.cnblogs.com/sddai/p/14646581.html） The derivation in , Besides , The author also realized pytorch Version of EMA.
stay paddle2 in EMA It can be divided into static plate and dynamic plate , The static plate is officially in paddle2.0 Implementation in , The following will be a brief introduction . The dynamic plate needs to be realized by itself , This will be implemented in the second section of this article . At the beginning of the training model ema In effect , Can make ema_model Very poor performance in the test set , Because there are few model updates at this stage , Most of its weights are just initialized . It is recommended to use after a certain period of training ema.

1、 Static plate EMA

Introduction to the official website https://www.paddlepaddle.org.cn/documentation/docs/zh/api/paddle/static/ExponentialMovingAverage_cn.html#exponentialmovingaverage

Import Can only be used for static graphs

from paddle.static import ExponentialMovingAverage

initialization </