On weight decay and discarding method

This article is the learning notes of Teacher Li Mu and teacher Wang Mu's video

Weight decline

The more parameters of a general model, the larger the capacity of the model （ The degree to which the model can fit the data ）, In order to prevent the model from over fitting, sometimes we need to reduce the capacity of the model , For example, by limiting the range of parameter values to achieve the purpose of reducing the model capacity .
$$minl(w,b)subjectto||w|{|}^2\le \theta$$

$\theta$ The smaller the regular term is, the smaller the regular term is


This is a constraint of $||w|{|}^2\le \theta$ Conditional extremum problem of , Use Lagrange multiplier method to solve , So structure ：
$$minl(w,b)+\frac{\lambda }{2}(||w|{|}^2-\theta )$$
And for $\lambda$ and $\theta$ Knowing one can relieve the other , So it can be equivalent to ：
$$minl(w,b)+\frac{\lambda }{2}||w|{|}^2$$
It can be proved that $\lambda \to \infty w^{\ast }\to 0$ Use $\lambda$ control $\theta$

As shown in the figure below $C$ representative $\theta$ The coordinate axes of the two directions are $w$ Size , so $C$ The bigger it is $w$ The smaller it is, the more it controls the capacity of the model .

Then the gradient calculation and parameter update become ：

gradient ：
$$\frac{\partial }{\partial w}(l(w,b)+\frac{\lambda }{2}||w|{|}^2)=\frac{l(w,b)}{\partial w}+\lambda w$$

After extracting the common factor, the parameters are updated and publicized ：
$$w_{t+1}=(1-\eta \lambda )w_t-\eta \frac{\partial l(w_t,b_t)}{\partial w_t}$$
When $\lambda \eta <1$ It is called weight decay .

The law of abandonment

Discard the output elements $x_i$ Do the following disturbance ：
$$\{\begin{array}{c}0,probabilityp\\ \frac{x_i}{1-p},otherise\end{array}$$
After this disturbance, for each $x_i$ Expectations remain $x_i$
$$E(x_i^{{}^{\prime }})=0\cdot p+(1-p)\cdot \frac{x_i}{1-p}=x_i$$

The discard method randomly sets the output of some hidden layers to 0, So as to control the complexity of the model , Its discard probability is a super parameter that controls the complexity of the model .

dropout Only enable during training , Used to adjust parameters , It is not used in reasoning .