[basic knowledge of deep learning - 39] comparison of BN, LN and WN

BN

• batch normalization, Make it normalize the distribution of data into the standard normal distribution before the activation function receives the input , Make the input value of the activation function fall in the input sensitive area , That is, areas with large gradients . So that the gradient does not disappear 、 Reduce training time .
• BN More suitable batch Larger scenes , And the data distribution should be relatively close .

WN

• weigit normalization, The main thing is to regularize the weight of the network , Make the network not too complicated , Don't weigh too much , Can prevent over fitting .

LN

• layer normalization, It is relative to the BN, Not dependent on the whole batch, Instead, it normalizes the input of a certain layer , More suitable for small batch、RNN、MLP Scene .

Bloggers will continue to update some basic knowledge related to in-depth learning, as well as problems and insights encountered in work , Please pay attention if you like 、 give the thumbs-up 、 Collection .