About snake equation (2)

Reference source ：(6 A private letter / 20 Bar message ) What is gradient descent method ？ - You know (zhihu.com) ------------ Ma classmate

Every gradient vector The module length is listed , It can be seen that it is decreasing , Therefore, this method is called gradient descent method ：

    

It's easy to understand , When finally tends to 0 From time to tome ：

So the minimum value obtained by gradient descent method （ Or nearby ）.

Step setting ：

As mentioned above, you can pass the step   To control the distance of each movement , Let's take a look at the impact of asynchronous length on the final result .

The step size is too small

  If set   It's too small , iteration 20 After that, it is still far from the bottom of the valley , actually 100 You can't reach the bottom after times .

                  

  Appropriate step size

  If set ,, Is a more appropriate step ,10 The minimum value is almost found once ：

The step size is too big

If set  ,  At this time, it will vibrate back and forth （ The figure below looks like there are only two points , In fact, back and forth between these two points ）：

       

  Sum up , Different steps As the number of iterations increases , Will result in optimized functions   Of Values vary :

 

  Find the right step , It's a craft job , The above picture can be drawn in the project , Manually adjust according to the image ：

  It's almost linear （ The blue line ） May be smaller , It needs to be raised

  Go up （ Red thread ）, Naturally Too big , Need to be lowered .

  At first, the decline was particularly rapid , Then there is little change （ Brown line ）, May be more , Need to be lowered .