Teacher lihongyi's notes on machine learning -4.2 batch normalization

notes ： This article is my study of Teacher Li Hongyi 《 machine learning 》 Course 2021/2022 The notes （ Course website ）, The pictures in the text are from the course PPT. Welcome to communicate and give more advice , thank you ！

Lecture 4.2 training tip: Batch Normalization

This lesson introduces a Deep Neural Network Of training tip, In the training CNN When it comes to .

So let's go back , Encounter the situation shown on the left of the following figure , Two parameters loss The curves are very different , A steep , A gentle . It is not easy to set up such training learning rate, It's hard to get to loss The lowest point . The previous lesson introduced adaptive learning rate Solutions for , According to different parameters gradient Different , Adjust accordingly learning rate.

This problem can also be changed landscape of error surface To solve . So let's analyze this , Why did this happen error surface, So you know how to adjust .

As shown in the figure above ,$w_1$ and $w_2$ Of loss The reason why the curves differ so much , Because its input $x_1$ and $x_2$ The magnitude of values varies greatly . If $w_1,w_2$ Make the same change $△w$, $x_1$ The value of is small , cause $y$ Small change in , And then $L$ The impact is also small , therefore $w_1$ The direction of the gradient Just small . and $x_2$ Value ratio of $x_1$ Two orders of magnitude larger , cause $y$ Great changes in . Accordingly , $w_2$ The direction of the gradient Big .

Through this analysis , We found that , Must let $w_1$ and $w_2$ The direction of the gradient Close （ As shown on the right of the figure above ）, You should make the corresponding input $x_1$ and $x_2$ Is in the same range . The corresponding method is feature normalization.

In this paper, feature normalization One way . The specific operation is ： hypothesis $x^1,x^2,...,x^R$ Represents a set of inputs samples, Put them in each dimension (feature) Subtract the corresponding mean from (mean), Divided by the corresponding variance (variance), As shown in the formula in the following figure . finish normalization after , Input samples The mean in each dimension is 0, The variance is 1.

My thinking ： Traditional machine learning methods , For example, similar operations will be performed on data in linear regression . However, in traditional machine learning, features are selected manually first , And then we do on the eigenvector Normalization Or normalization of maximum and minimum values . Deep learning is a feature of automatic machine learning , I don't know how the model will select or combine each dimension vector to generate features , Therefore, it is done in all dimensions Normalization.

doubt ： about DNN The Internet , Input is Vector, Do it Normalization A good understanding , That is, each dimension of the input vector . and CNN The network input is an image , In which dimension do you do ？ Is in depth (channel) On , Such as color images , Is in the 3 individual channel Find the mean value respectively 、 variance , Do it again Normalization. Recommend This article Example , Demonstrate the calculation process once , Be clear about .

Besides , stay Deep Learning in , Also need to consider hidden layer （ Middle layer ） Output . As shown in the figure below $z^i$ , It is also the input to the next level , So we also need to do Normalization.

Where to do Normalization Well ？

Generally speaking , stay activation function Before （ Yes $z^i$ do ） Or after （ Yes $a^i$ do ） Fine . however , If activation function yes Sigmoid, because Sigmoid function In the value range [0,1] Of gradient more , So right. $z^i$ do normalization Better .

Yes $z^i$ Did normalization after , There is an interesting phenomenon ： Originally, each piece of data （ each sample ） They don't affect each other . for example $z^1$ The change of only affects $a^1$ . But do normalization In all data (samples) Find the mean and variance , As shown in the figure below ,$z^1$ The change will be through $\mu ,\sigma$ influence ${\tilde{z}}^2,{\tilde{z}}^3$, Which in turn affects $a^2,a^3$.

in other words , It used to be that you only had to consider one at a time sample , Input to deep neural network Get one output. do Normalization Will make samples They influence each other , So now we need to consider one batch at a time samples, Enter a large deep neural network, Get a batch of outputs.

In practice, the data of all training sets is very large , do Normalization Think about one batch Of samples, So called Batch Normalization. Be careful ：batch size Very small , For example, equal to 1 when , Can not find the mean and variance , It makes no sense .Batch Normalization Apply to batch size When I was big .

Put the input samples All dimensions of are adjusted to mean 0, The variance of 1 The distribution of , Sometimes it may not be easy to solve the problem , At this point you can make a little change . As shown in the formula in the upper right corner of the figure below . Be careful ：$\gamma$ and $\beta$ The value of is obtained by learning , That is, adjust according to the model and data .$\gamma$ The initial setting is all 1 vector ,$\beta$ The initial setting is all 0 vector .

Batch Normalization Is good , But in testing I had a problem when I was .testing Or actual deployment , Data is processed in real time , It is impossible to wait for one at a time batch To deal with .

What shall I do? ？

During training , Every time I deal with one batch, Just follow the formula shown in the figure below , Update mean $\mu$ Of moving average $\overline{\mu }$ ,$\sigma$ Similar operation . After training , Take what you get $\overline{\mu }$ and $\overline{\sigma }$ Used as a testing data The mean and variance of .

Why do you say Batch Normalization It is helpful for optimizing parameters ？

“Experimental results (and theoretically analysis) support batch normalization change the landscape of error surface.”

Experimental results and theoretical analysis verify Batch Normalization By changing error surface, Give Way optimization Better .

If you think this article is good , Please praise and support , thank you ！

Pay attention to me Ning Meng Julie, learn from each other , Communicate more ！

Read more notes , Please click on Mr. Li Hongyi 《 machine learning 》 note – Catalogue .

Reference resources

1. Mr. Li Hongyi 《 machine learning 2022》：

Course website ：https://speech.ee.ntu.edu.tw/~hylee/ml/2022-spring.php

video ：https://www.bilibili.com/video/BV1Wv411h7kN

2.Batch Normalization Examples of operations ：https://blog.csdn.net/weixin_42211626/article/details/122854079