17. Attenuation of momentum and learning rate

momentum

It can be understood as inertia ,

Subtracting a gradient means going in the direction of the gradient , At this point, subtract one zk, Means to zk Go in the right direction
zk Is a superposition operator , Represents the direction of the last gradient ▽w(k-1),

adam Itself is the optimization made by using momentum , There is no need to manage this variable extra .

optimizer=torch.optim.SGD(model.parameters(),args.lr,
                          momentum=args.momentum,
                          weight_decay=args.weight_decay)

scheduler=ReduceLROnPlateau(optimizer,'min')

for epoch in xrange(args.start_epoch,args.epochs):
    train(train_loader,model,criteon,optimizer,epoch)    		      	 
    result_avg,loss_val=
    validate(val_loader,model,criteon,epoch)
    scheduler.step(loss_val)

learning rate tunning Attenuation of learning rate

We will force the learning rate to slowly drop to 0

programme 1：

scheduler=ReduceLROnPlateau(optimizer,'min')

loss Gentle after descent , A straight line no longer decreases , It's about reducing lr. Parameters patience,patience Next time , monitor loss, Find out loss There's no reduction , Will reduce lr, Every time lr Member reduction factor （ for example 0.5 or 0.08）.

scheduler.step(loss_val)

Listen every time you call loss, Judge loss Whether it will not decrease for a long time , If it is , Reduce according to the rules lr, If not, do nothing , It only serves as a record .

programme 2：

Every provision x individual epoch, It will fade a little lr.

scheduler=StepLR(optimizer,step_size=30,gamma=0.1)

Every time 30 individual epoch,lr=lr*gamma
Usually set to 1k