Why is gradient the fastest changing direction of function

Click on the above “ Xiaobai studies vision ”, Optional plus " Star standard " or “ Roof placement ”

 Heavy dry goods , First time delivery 

Preface

Most problems in machine learning are optimization problems , Most optimization problems can be solved by gradient descent method . This paper explains in detail several important concepts that are easy to be confused in high numbers , Such as the difference between derivative and differential , The concept of partial derivative , Relationship between directional derivative and gradient , If you fully grasp these concepts , We can well understand why the gradient is the fastest changing direction of the function .

Catalog

---

1、 Derivative and differential

2、 Partial derivative

3、 Relationship between directional derivative and gradient

4、 summary

Derivative and differential

Definition of derivative

The essence ： Derivative describes the trend of the change speed of a function at a point , It's a rate of change . For example, the derivative of the curve equation is the slope changing with the point , The derivative of the equation of motion is the rate of change with time .

The definition of differential

The essence ： Differentiation describes the change of a function from one point to another infinitesimal point .

The relationship between function increment and differential

This section analyzes the relationship between function increment and differentiation from the perspective of graphics and algebra ：

• Graphic angle ：

As shown in the figure above , function f(x) stay M The derivative at point is a straight line T The slope of tanα,Δy yes M Point move Δx Function increment when ,dy Is the function relative to Δx Differential of .

When Δx->0 when ,,.

• Algebraic angle ：

Partial derivative

Partial derivative is the derivative of a function relative to an axis , Other axial directions are assumed to be constant , If we consider binary variables f(x,y), Partial derivatives are defined as follows ：

partial The geometric meaning of the derivative

Relationship between directional derivative and gradient

Directional derivative

Let's explain the directional derivative by discussing the graph of partial derivative . Let the surface equation z=f(x,y) Projection to XY Plane , Get the projection plane , Here's the picture ：

M1 by M0 stay XY Projection point of the face , It can be seen from the above figure , There are countless straight lines passing by M1 spot , These lines represent the direction , We think of surfaces M1 The directional derivative of a point is the derivative of these straight lines ,M1 The directional derivatives of points are also infinite , We use variables α To represent straight lines in different directions .

gradient

Gradient is a vector , The gradient of each point on the surface is constant ,P0 The gradient of the point is as follows ：

Relationship between directional derivative and gradient

Find the surface above M0 in P Gradient and directional derivative of points

The unit vectors of gradient and directional derivative are shown in the following two figures ：

Translation gradient vector , Intersect it with the unit vector of the directional derivative , The included angle is θ, Here's the picture ：

The red line represents the gradient , Blue represents the unit vector of directional derivative , Take the inner product of these two vectors , have to ：

Conclusion ： There are countless directional derivatives at the midpoint of the surface , When the directional derivative is consistent with the gradient direction , The derivative value gets the maximum , Equivalent to that the point has the fastest change value in the gradient direction . The gradient direction is the direction in which the value of the function increases fastest , The opposite direction of the gradient is the direction in which the value of the function decreases fastest .

summary

This paper introduces several confusing concepts in advanced mathematics textbooks , It is proved that when the direction derivative is consistent with the gradient direction by combining the graphic method and formula derivation , The function value changes the fastest . therefore , Machine learning often uses gradient method to solve optimization problems .

Reference resources

《 Advanced mathematics 》 The seventh edition , Tongji University

The good news ！

Xiaobai learns visual knowledge about the planet

Open to the outside world

 download 1：OpenCV-Contrib Chinese version of extension module 

 stay 「 Xiaobai studies vision 」 Official account back office reply ： Extension module Chinese course , You can download the first copy of the whole network OpenCV Extension module tutorial Chinese version , Cover expansion module installation 、SFM Algorithm 、 Stereo vision 、 Target tracking 、 Biological vision 、 Super resolution processing and other more than 20 chapters .


 download 2：Python Visual combat project 52 speak 
 stay 「 Xiaobai studies vision 」 Official account back office reply ：Python Visual combat project , You can download, including image segmentation 、 Mask detection 、 Lane line detection 、 Vehicle count 、 Add Eyeliner 、 License plate recognition 、 Character recognition 、 Emotional tests 、 Text content extraction 、 Face recognition, etc 31 A visual combat project , Help fast school computer vision .


 download 3：OpenCV Actual project 20 speak 
 stay 「 Xiaobai studies vision 」 Official account back office reply ：OpenCV Actual project 20 speak , You can download the 20 Based on OpenCV Realization 20 A real project , Realization OpenCV Learn advanced .


 Communication group 

 Welcome to join the official account reader group to communicate with your colleagues , There are SLAM、 3 d visual 、 sensor 、 Autopilot 、 Computational photography 、 testing 、 Division 、 distinguish 、 Medical imaging 、GAN、 Wechat groups such as algorithm competition （ It will be subdivided gradually in the future ）, Please scan the following micro signal clustering , remarks ：” nickname + School / company + Research direction “, for example ：” Zhang San  +  Shanghai Jiaotong University  +  Vision SLAM“. Please note... According to the format , Otherwise, it will not pass . After successful addition, they will be invited to relevant wechat groups according to the research direction . Please do not send ads in the group , Or you'll be invited out , Thanks for your understanding ~