Learning rate optimization strategy

Warmup Strategy

Deep learning , The initial weight of the model is randomly generated in the training start-up stage , choice Warmup The strategy can make the model use less learning at the beginning of training
Practice at a regular rate , After a set number of iterations , After the model tends to be stable , Then change to the preset learning rate , Achieve the effect of preheating the learning rate , It can prevent the model from shaking , Accelerate the convergence speed of the network , Improve the effect .

Used in experiments Warmup In strategy Gradual Warmup, That is, in the preheating stage of learning rate, the learning rate will gradually increase with the increase of iteration times , Until the end of the warm-up phase, the learning rate reaches the preset value , And then do the follow-up training , This can avoid the sudden increase of learning rate and the sharp increase of training error .

Poly Strategy

Learning rate is a super parameter that has a great influence on the weight update of the model . Only when the initial learning rate is set reasonably can the model be optimized , Too small will lead to slow convergence , Too large will lead to instability or convergence failure . The learning rate needs to change with the degree of online training , Its change strategy is very important , There are many strategies in deep learning , Such as Fixed Strategy 、Poly Strategy and sigmoid Strategy . In this paper, the experimental results are as follows SGD The optimization strategy adds Poly Learning rate decay strategy , The current learning rate is