Tsinghua University product: penalty gradient norm improves generalization of deep learning model

1 introduction

  The structure of neural network is simple , Insufficient training sample size , It will lead to the low classification accuracy of the trained model ; The structure of neural network is complex , Training sample size is too large , It will lead to over fitting of the model , Therefore, how to train neural network to improve the generalization of model is a very core problem in the field of artificial intelligence . I recently read an article related to this problem , In this paper, the author improves the generalization of the deep learning model by adding the constraint of the gradient norm of the regularization term in the loss function . The author expounds and verifies the methods in this paper in detail from two aspects of principle and experiment .$\mathrm{L}\mathrm{i}\mathrm{p}\mathrm{s}\mathrm{c}\mathrm{h}\mathrm{i}\mathrm{t}\mathrm{z}$ Continuous learning is a very important and common mathematical tool in the theoretical analysis of deep learning , This paper is based on neural network loss function $yes\mathrm{L}\mathrm{i}\mathrm{p}\mathrm{s}\mathrm{c}\mathrm{h}\mathrm{i}\mathrm{t}\mathrm{z}$ Mathematical derivation with continuity as the starting point . In order to facilitate readers to more smoothly appreciate the author's beautiful mathematical proof ideas and processes , This paper supplements the details of mathematical proof that is not carried out in the paper .

Thesis link ：https://arxiv.org/abs/2202.03599

2 $\mathrm{L}\mathrm{i}\mathrm{p}\mathrm{s}\mathrm{c}\mathrm{h}\mathrm{i}\mathrm{z}$ continuity

  Given a training data set $\mathcal{S}=\{(x_i,y_i){\}}_{i=0}^n$ Obey the distribution $\mathcal{D}$, One with parameters $\theta \in \Theta$ The neural network of $f(\cdot ;\theta )$, The loss function is $$L_{\mathcal{S}}=\frac{1}{N}\sum _{i=1}^{N}l(\widehat{y_i,y_i,\theta })$$ When it is necessary to constrain the gradient norm in the loss function , Then there is the following loss function $$L(\theta )=L_{\mathcal{S}}+\lambda \cdot \Vert {\nabla }_{\theta }{L}_{\mathcal{S}}(\theta ){\Vert }_p$$ among $\Vert \cdot {\Vert }_p$ Express $p$ norm ,$\lambda \in {\mathbb{R}}^{+}$ Is the gradient penalty coefficient . In general , The loss function introduces the regularization term of gradient, which will make it have smaller local in the optimization process $\mathrm{L}\mathrm{i}\mathrm{p}\mathrm{s}\mathrm{c}\mathrm{h}\mathrm{i}\mathrm{t}\mathrm{z}$ constant ,$\mathrm{L}\mathrm{i}\mathrm{p}\mathrm{s}\mathrm{c}\mathrm{h}\mathrm{i}\mathrm{t}\mathrm{z}$ The smaller the constant , That means the smoother the loss function , The smooth area of the flat loss function is easy to optimize the weight parameters of the loss function . Further, it will make the trained deep learning model have better generalization .
  A very important and common concept in deep learning is $\mathrm{L}\mathrm{i}\mathrm{p}\mathrm{s}\mathrm{c}\mathrm{h}\mathrm{i}\mathrm{t}\mathrm{z}$ continuity . Given a space $\Omega \subset {\mathbb{R}}^n$, For the function $h:\Omega \to {\mathbb{R}}^m$, If there is a constant $K$, about $\forall {\theta }_1,{\theta }_2\in \Omega$ If the following conditions are met, it is called $\mathrm{L}\mathrm{i}\mathrm{p}\mathrm{s}\mathrm{c}\mathrm{h}\mathrm{i}\mathrm{t}\mathrm{z}$ continuity $$\Vert h({\theta }_1)-h({\theta }_2){\Vert }_2\le K\cdot \Vert {\theta }_1-{\theta }_2{\Vert }_2$$ among $K$ It means $\mathrm{L}\mathrm{i}\mathrm{p}\mathrm{s}\mathrm{c}\mathrm{h}\mathrm{i}\mathrm{t}\mathrm{z}$ constant . If for parameter space $\Theta \subset \Omega$, If $\Theta$ There is a neighborhood $\mathcal{A}$, And $h{|}_{\mathcal{A}}$ yes $\mathrm{L}\mathrm{i}\mathrm{p}\mathrm{s}\mathrm{c}\mathrm{h}\mathrm{i}\mathrm{t}\mathrm{z}$ continuity , said $h$ It's local $\mathrm{L}\mathrm{i}\mathrm{p}\mathrm{s}\mathrm{c}\mathrm{h}\mathrm{i}\mathrm{t}\mathrm{z}$ continuity . Intuitive to see ,$\mathrm{L}\mathrm{i}\mathrm{p}\mathrm{s}\mathrm{c}\mathrm{h}\mathrm{i}\mathrm{t}\mathrm{z}$ A constant describes an upper bound of the output with respect to the rate of change of the input . For a small $\mathrm{L}\mathrm{i}\mathrm{p}\mathrm{s}\mathrm{c}\mathrm{h}\mathrm{i}\mathrm{t}\mathrm{z}$ Parameters , In the neighborhood $\mathcal{A}$ Any two points given in , Their output changes are limited to a small range .
  According to the differential mean value theorem , Given a minimum point ${\theta }_i$, For any point $\forall {\theta }_i^{\prime }\in \mathcal{A}$, Then the following formula holds $$\Vert |L({\theta }_i^{\prime })-L({\theta }_i){\Vert }_2=\Vert \nabla L(\zeta )({\theta }_i^{\prime }-{\theta }_i){\Vert }_2$$ among $\zeta =c{\theta }_i+(1-c){\theta }_i^{\prime },c\in [0,1]$, according to $\mathrm{C}\mathrm{a}\mathrm{u}\mathrm{c}\mathrm{h}\mathrm{y}\text{-}\mathrm{S}\mathrm{c}\mathrm{h}\mathrm{w}\mathrm{a}\mathrm{r}\mathrm{z}$ We can see that $$\Vert |L({\theta }_i^{\prime })-L({\theta }_i){\Vert }_2\le \Vert \nabla L(\zeta ){\Vert }_2\Vert ({\theta }_i^{\prime }-{\theta }_i){\Vert }_2$$ When ${\theta }_i^{\prime }\to \theta$ when , Corresponding $\mathrm{L}\mathrm{i}\mathrm{p}\mathrm{s}\mathrm{c}\mathrm{h}\mathrm{i}\mathrm{z}$ Constant approach $\Vert \nabla L({\theta }_i){\Vert }_2$. Therefore, we can reduce $\Vert \nabla L({\theta }_i)\Vert$ The numerical value of makes the model converge more smoothly .

3 Paper method

  The gradient of the loss function with gradient norm constraint can be obtained
$${\nabla }_{\theta }L(\theta )={\nabla }_{\theta }{L}_{\mathcal{S}}(\theta )+{\nabla }_{\theta }(\lambda \cdot \Vert {\nabla }_{\theta }{L}_{\mathcal{S}}(\theta ){\Vert }_p)$$ In this paper , The author made $p=2$, At this time, there is the following derivation process $$\begin{array}{rl}{\nabla }_{\theta }\Vert {\nabla }_{\theta }{L}_{\mathcal{S}}(\theta ){\Vert }_2& ={\nabla }_{\theta }[{\nabla }_{\theta }^{\top }{L}_{\mathcal{S}}(\theta )\cdot {\nabla }_{\theta }{L}_{\mathcal{S}}(\theta ){]}^{\frac{1}{2}}\\ & =\frac{1}{2}\cdot {\nabla }_{\theta }^2{L}_{\mathcal{S}}(\theta )\frac{{\nabla }_{\theta }{L}_{\mathcal{S}}(\theta )}{\Vert {\nabla }_{\theta }{L}_{\mathcal{S}}(\theta ){\Vert }_2}\end{array}$$ This result is brought into the loss function of gradient norm constraint , Then there is the following formula
$${\nabla }_{\theta }L(\theta )={\nabla }_{\theta }{L}_{\mathcal{S}}(\theta )+\lambda \cdot {\nabla }_{\theta }^2{L}_{\mathcal{S}}(\theta )\frac{{\nabla }_{\theta }{L}_{\mathcal{S}}(\theta )}{\Vert {\nabla }_{\theta }{L}_{\mathcal{S}}(\theta ){\Vert }_2}$$ You can find , The above formula involves $\mathrm{H}\mathrm{e}\mathrm{s}\mathrm{s}\mathrm{i}\mathrm{a}\mathrm{n}$ Matrix calculation , In deep learning , To calculate the parameters of $\mathrm{H}\mathrm{e}\mathrm{s}\mathrm{s}\mathrm{i}\mathrm{a}\mathrm{n}$ Matrix will bring high computational cost , So we need to use some approximate methods . The author expands the loss function by Taylor expansion , Among them, the order is $H={\nabla }_{\theta }^2{L}_{\mathcal{S}}(\theta )$, Then there are $$L_{\mathcal{S}}(\theta +\Delta \theta )=L_{\mathcal{S}}(\theta )+{\nabla }_{\theta }^{\top }{L}_{\mathcal{S}}(\theta )\cdot \Delta \theta +\frac{1}{2}\Delta {\theta }^{\top }H\Delta \theta +\mathcal{O}(\Vert \Delta \theta {\Vert }_2^2)$$ Then there are $$\begin{array}{rl}{\nabla }_{\theta }{L}_{\mathcal{S}}(\theta +\Delta \theta )& ={\nabla }_{\Delta \theta }{L}_{\mathcal{S}}(\theta +\Delta \theta )={\nabla }_{\theta }{L}_{\mathcal{S}}(\theta )+H\Delta \theta +\mathcal{O}(\Vert \Delta \theta {\Vert }_2^2)\end{array}$$ Among them, the order is $\Delta \theta =rv$,$r$ Represents a small number ,$v$ It's a vector , If you bring in the above formula, you have $$Hv=\frac{{\nabla }_{\theta }{L}_{\mathcal{S}}(\theta +rv)-{\nabla }_{\theta }{L}_{\mathcal{S}}(\theta )}{r}+\mathcal{O}(r)$$ If you make $v=\frac{{\nabla }_{\theta }{L}_{\mathcal{S}}(\theta )}{\Vert {\nabla }_{\theta }{L}_{\mathcal{S}}(\theta )\Vert }$, Then there are $$H\frac{{\nabla }_{\theta }{L}_{\mathcal{S}}(\theta )}{\Vert {\nabla }_{\theta }{L}_{\mathcal{S}}(\theta ){\Vert }_2}\approx \frac{{\nabla }_{\theta }L(\theta +r\frac{{\nabla }_{\theta }{L}_{\mathcal{S}}(\theta )}{\Vert {\nabla }_{\theta }{L}_{\mathcal{S}}(\theta ){\Vert }_2})-{\nabla }_{\theta }L(\theta )}{r}$$
in summary , After finishing, you can get
$$\begin{array}{rl}{\nabla }_{\theta }L(\theta )& ={\nabla }_{\theta }{L}_{\mathcal{S}}(\theta )+\frac{\lambda }{r}\cdot ({\nabla }_{\theta }{L}_{\mathcal{S}}(\theta +r\frac{{\nabla }_{\theta }{L}_{\mathcal{S}}(\theta )}{\Vert {\nabla }_{\theta }{L}_{\mathcal{S}}(\theta ){\Vert }_2})-{\nabla }_{\theta }{L}_{\mathcal{S}}(\theta ))\\ & =(1-\alpha ){\nabla }_{\theta }{L}_{\mathcal{S}}(\theta )+\alpha {\nabla }_{\theta }{L}_{\mathcal{S}}(\theta +r\frac{{\nabla }_{\theta }{L}_{\mathcal{S}}(\theta )}{\Vert {\nabla }_{\theta }{L}_{\mathcal{S}}(\theta ){\Vert }_2})\end{array}$$ among $\alpha =\frac{\lambda }{r}$, call $\alpha$ Is the equilibrium coefficient , The value range is $0\le \alpha \le 1$. In order to avoid when calculating the gradient approximately , The gradient of the second Necklace rule in the above formula needs to be calculated $\mathrm{H}\mathrm{e}\mathrm{s}\mathrm{s}\mathrm{i}\mathrm{a}\mathrm{n}$ matrix , After making the following approximation, there is $${\nabla }_{\theta }{L}_{\mathcal{S}}(\theta +r\frac{{\nabla }_{\theta }{L}_{\mathcal{S}}(\theta )}{\Vert {\nabla }_{\theta }{L}_{\mathcal{S}}(\theta ){\Vert }_2})\approx {\nabla }_{\theta }{L}_{\mathcal{S}}(\theta ){|}_{\theta =\theta +r\frac{{\nabla }_{\theta }{L}_{\mathcal{S}}(\theta )}{\Vert {\nabla }_{\theta }{L}_{\mathcal{S}}(\theta ){\Vert }_2}}$$ The following algorithm flow chart summarizes the training methods of this paper

4 experimental result

  The following table shows in $\mathrm{C}\mathrm{i}\mathrm{f}\mathrm{a}\mathrm{r}10$ and $\mathrm{C}\mathrm{i}\mathrm{f}\mathrm{a}\mathrm{r}100$ These two datasets are different $\mathrm{C}\mathrm{N}\mathrm{N}$ Network structure in standard training ,$\mathrm{S}\mathrm{A}\mathrm{M}$ Comparison of test error rate between the three training methods with gradient constraint in this paper . You can intuitively find , The method proposed in this paper has the lowest test error rate in most cases , This also verifies from the side that the training of the paper method can improve $\mathrm{C}\mathrm{N}\mathrm{N}$ Generalization of model .

  The author of the paper is also in the current very popular network structure $\mathrm{V}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}\mathrm{T}\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{s}\mathrm{f}\mathrm{o}\mathrm{r}\mathrm{m}\mathrm{e}\mathrm{r}$ Experiments were carried out . The following table shows in $\mathrm{C}\mathrm{i}\mathrm{f}\mathrm{a}\mathrm{r}10$ and $\mathrm{C}\mathrm{i}\mathrm{f}\mathrm{a}\mathrm{r}100$ These two datasets are different $\mathrm{V}\mathrm{i}\mathrm{T}$ Network structure in standard training ,$\mathrm{S}\mathrm{A}\mathrm{M}$ Comparison of test error rate between the three training methods with gradient constraint in this paper . Similarly, it can also be found that the test error rate of the method proposed in this paper is the lowest in all cases , This shows that the method in this paper can also be mentioned $\mathrm{V}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{s}\mathrm{f}\mathrm{o}\mathrm{r}\mathrm{m}\mathrm{e}\mathrm{r}$ Generalization of model .