Weight decay

Weight falloff Use the mean square norm as a hard limit , adopt Limit parameter values w The scope of the To control the model capacity .dropout It is through Reduce parameters Value to simplify the model , Both can prevent over fitting .

Hyperparameters Control the importance of regularity , The intensity of punishment , =0 No punishment ; The bigger it is , The smaller the parameter value is controlled .

 

 

  Penalty item Is a circle centered on the origin , Loss item Such as the right green ring , The best advantage of distance loss function Closer , The smaller the gradient transformation , namely w Change to loss Less impact (ABC The gradient gradually decreases ). So find one Balance point , Penalty item Plus loss items Minimum , by The optimal solution of the new objective function .