[foundation of deep learning] back propagation method (1)

We have already learned the learning process of neural network , The minimum value of the cost function is approached by the gradient descent method , And then continuously update the weight , Get the best parameters .

however , The previous gradient calculation , The method of central difference is adopted , It's complicated to calculate , Now let's start to learn an efficient gradient calculation method —— Back propagation .

The concept of computational graph

Bought... In the supermarket 2 individual 100 An apple in yen , The excise tax is 10%, Calculate payment amount .

We use graphics , Describe the calculation process of the payment amount .

Put it another way ：

We found that , In solving this problem , Our own ideas and graphics are exactly the same ！

1. Calculate the value according to the unit price and quantity of apples .

2. According to tax rate , Calculate the amount to be paid .

How do we calculate the derivative of the payment amount with respect to the apple unit price ？

You can use the reverse method of just calculating the payment amount , Calculate the corresponding derivative （ How to calculate , Later on ）. We found that , The calculation diagram is passed forward , You can calculate the payment amount step by step . And through back propagation , You can exit the derivative of the payment amount to a certain variable step by step .

Calculation f(x,y) = (x+y)^2 Yes x Partial derivative of

First , Let's change the function ：

  According to the derivation formula ,

We refer to step , The process of step-by-step calculation of partial derivatives is completely consistent with the back-propagation of the calculation diagram .

Back propagation of additive computation

Neural network , Addition calculation and multiplication calculation are common , Let's take a look at .

 

For addition calculation , We found that , Their partial derivatives are 1.

 

Back propagation of multiplication

For multiplication calculation ？

  We find that the back propagation of multiplication is the product of other terms ：