In depth learning report (3)

Catalog

Chapter six initialization

Chapter vii. Parameter adjustment

---

Chapter six initialization

One 、 Why initialize ？

• Selection of initial point , Sometimes it can decide whether the algorithm converges
• When converging , The initial point can determine how fast the learning converges , Whether it can converge to a high or low cost point
• Too large initialization leads to gradient explosion , Too small initialization causes the gradient to disappear

Two 、 What is good initialization ？

• So that the activation value of each neuron layer will not be saturated
• The activation value of each layer cannot be 0

3、 ... and 、 Common initialization

1、 All zero initialization

• All zero initialization ： The initial value of the parameter is 0.
• shortcoming ： Neurons in the same layer will learn the same characteristics , The symmetry of different neurons cannot be destroyed . If the weights of neurons are initialized to 0 , The output of all neurons will be the same , In addition to output , The value of all nodes in the middle tier is zero . General neural networks have a symmetrical structure , Then when the first error back propagation is carried out , The updated network parameters will be the same , At the next update , Learning the same network parameters can not extract useful features , Therefore, deep learning models will not be used 0 Initialize all parameters .

2、 Random initialization

• Random initialization ： Initialize the parameter to a small random number . Generally, the random value is μ , The standard deviation is σ Sampling in Gaussian distribution , Each dimension of the final parameter comes from a multidimensional Gaussian distribution .
• shortcoming : Once the random distribution is chosen improperly , Will lead to network optimization in trouble . If the initial value of the parameter is too small , In back propagation, it will lead to small gradients , For deep Networks , There will be gradient dispersion problem , Reduce the convergence speed of parameters . If the initial value of the parameter is too large , Then neurons will be easily saturated .

3、Xavier initialization

 4、He initialization

---

Chapter vii. Parameter adjustment

1、 Trial and error method ： For example, after students design an experiment , Follow all the steps of the learning process （ Visualization from data collection to feature map ）, Then iterate on the super parameter in turn until the time ends .

2、 The grid search ： If there are three or fewer super parameters , The common super parameter search method is grid search . For each super parameter , The user selects a smaller set of finite values to explore . then , The Cartesian product of these hyperparameters yields a set of hyperparameters , Grid search uses each set of hyperparametric training models . Select the super parameter with the smallest error in the verification set as the best super parameter .

3、 Random search ： The only difference between grid search and random search is the first step of the policy cycle , Random search randomly selects points in the configuration space .

4、 Bayesian optimization ：

• Build the model
• Select the super parameter
• Training , assessment
• Optimization model , Return to step 2