Machine learning and deep learning -- normalization processing

One 、 Normalization in machine learning

1、 Normalization

（1） After normalization, the speed of gradient descent to find the optimal solution is accelerated
（2） Normalization has the potential to improve accuracy

Detailed explanation ：
（1） After normalization, the speed of gradient descent to find the optimal solution is accelerated

\qquad a. The left figure is the contour line without normalization , It can be seen from the left figure that there is a big difference between the two feature change ranges , The contour line formed is very sharp , When the gradient descent method is used to find the optimal solution , It's very likely to go “ And ” Font route , So it needs many iterations to converge ;
\qquad b. Right picture , Normalize the data , At this time, the contour lines of the two features appear smooth , In the process of gradient descent, it can converge faster ;
\qquad Therefore, if the machine learning model uses gradient descent to find the optimal solution , Normalization is very necessary , Otherwise, it will take more iterations to converge or even fail to converge .

（2） Normalization may improve accuracy
Some classifiers need to calculate the distance between samples （ For example, European distance ）, for example KNN. If a range of eigenvalues is very large , Then the calculation of distance depends on this feature , But it may not conform to the actual situation , For example, the characteristics of small cure range may be more important in the actual situation .

2、 Normalization method

（1） Linear normalization
（2） Normalization of standard deviation ： Characteristics minus mean divided by variance
（3） Nonlinear normalization

Two 、 In depth learning BN layer

1.BN layer

\qquad BN,Batch Normalization（ Normalized layer ）
\qquad BN Problems solved by layer ： In deep neural networks , Each layer will operate on the data , Even if the data is normalized initially , But with the deepening of neural network layers , The distribution of data is likely to change dramatically , At this time, it brings difficulty to the training of network model .
\qquad At this point, you need to join BN layer , Make the data keep the same distribution in the process of deep neural network training .
\qquad In short ：BN Layer is to deal with data in a normative way , Make the distribution of data input values of each layer smooth , For example, the average value is 0, The variance of 1 Is a normal distribution .

2.BN Layer action

（1） Speed up the training and convergence of the network
（2） Prevent gradient disappearance and gradient explosion
（3） Prevent over fitting

Refer to the post