Automatic derivation of pytorch

1. Basic concepts

1.1 requires_grad

   If you need to derive a tensor , Then the value must be assigned during initialization requires_grad=True, for example

a = torch.tensor(2.3, requires_grad=True)

1.2 Calculation chart

   In the calculation chart , Data is represented by ellipses , Operations such as addition, subtraction, multiplication and division are represented by a rectangle . The data and operations are represented as a binary tree structure through the calculation diagram .

1.3 Leaf node

   In the calculation chart , The data created by users themselves is called leaf nodes , Like the one above w, x, b, It can also be said that the data that has not been calculated is leaf nodes . It can be used a.is_leaf( Note that attributes , Not the way ) Determine whether it is a leaf node .

print(a.is_leaf)

1.4 grad_fn

  z.grad_fn The output is <AddBackward0 at 0x8ea34d2cd342>, It means z Corresponding Direct operation .

  tensor Obtained by an operation , stay PyTorch The back propagation function of each operation is already defined , such as z By add That is, the result of the addition operation , that z.grad_fn What you get is add Back propagation function of function （ Derivative function ）. Notice what we get is AddBackward0 Behind a 0, It shows that a calculation diagram can appear many times add, Every add The back propagation function of is different .

1.5 next_functions

  z.grad_fn.next_functions The output is
(<AccumulateGrad at 0x7fb73c7cdad0>, 0L))
((<MulBackward0 at 0x7fb73c7cd7d0>, 0L),

  z By add The result of the operation is , that add The input to the operation is b and y, The output is b.grad_fn and y.grad_fn.

  AccumulateGrad What is it? ？a.grad What is it? ？ Why should the gradient be set 0？
about y.grad_fn We can know MulBackward It is the back propagation function corresponding to the product operation .

  b Just a leaf node , It's a Tensor, its grad_fn namely Accumlate_Grad Express this b The derivative of is cumulative . For example, the first time you spread in the direction , We come to the conclusion that b The derivative of is 3, namely a.grad by 3, But you take the derivative again , You will find a.grad by 6, This is called cumulative . So in Pytorch Inside , every last batch That is, the gradient will be reduced before each back propagation, that is grad All are set for 0.

1.6 retain_graph=True backward()

  z.backward(retain_graph=True)

  z.backward() From z The answer is z The derivative of each variable .

  retain_graph=True Means to save intermediate variables . Let's calculate z Yes w The derivative of is found to be y, Pay attention to this y In the calculation diagram of our example above, it is not specified by ourselves , It's from the middle , For the first time z.backward() seek z Yes w The derivative of will take y Value . But if we want to spread it once after this spread , So it's a mistake , Because a gradient game will automatically release the computing things in the middle , That is, there was no... In the second transmission y 了 , Unless you spread it forward again . So we can specify this in advance to ensure that the intermediate variable still exists after the first propagation .

1.7 hook function

   The derivation of non leaf nodes will be released , If you want to see its derivative , It can be used autograd.grad perhaps hook function .

2. summary

   About autograd, What we need to know is that we can create tensor When it's time to specify requires_grad = True Make differentiable , Then in the final function, use z.backward(). use a.grad View derivatives （ gradient ）.