Li Hongyi, machine learning 5. Tips for neural network design

One 、 Local minimum and saddle point

Optimization Fails because …

• local minima（ Local minimum ）
• saddle point（ saddle point ）

▲ Optimization failure reason

Two 、 Batch and momentum

• Optimization with Batch

▲ Batch optimization

• Gradient Descent + Momentum
  

▲ The gradient with momentum falls

3、 ... and 、 Automatically adjust the learning rate

RMSProp(root mean square prop, Root mean square )
Learning principles ∶ The attenuation factor is introduced on the basis of adaptive gradient , When the gradient accumulates , Would be right “ In the past ” And “ Now? ” Make a balance , Adjust the attenuation through super parameters .
Suitable for dealing with non-stationary targets （ That is, time related ), about RNN The effect is very good .

▲ RMSProp

Adam(Adaptive momentum optimization, Adaptive momentum optimization )
It is the most popular optimization method in deep learning , It combines the advantages that adaptive gradient is good at dealing with sparse gradient and root mean square is good at dealing with non-stationary targets , It is suitable for large data sets and high-dimensional space .

▲ Adam

Four 、 The effect of loss function

Mainly the impact on classification tasks .

▲ Loss of Classification

5、 ... and 、 Batch standardization

Batch standardization （Batch Normalization）, Change the range of different features （changing landscape）.

▲ Changing Landscape

▲ Feature Normalization

6、 ... and 、 summary

Datawhale Team learning , Li Hongyi 《 machine learning 》Task5. Tips for neural network design（ Neural network design skills ）. Including local minimum and saddle point 、 Batch and momentum 、 Automatically adjust the learning rate 、 Effect of loss function and batch standardization . Mr. Li clearly explained the principles of mathematics in his course , However, the emphasis on application can be ignored , Know the design skills .

——END——