Deep Learning with Pytorch - autograd

Deep Learning with Pytorch: A 60 Minute Blitz

Autograd: automatic differentiation

Autograd: Automatic differentiation

stay Pytorch The center of all neural networks in is autograd package
Let's first take a brief look at , Then we will train our first neural network .

autograd Package provides Tensors Automatic differentiation of all operations on . This is a runtime defined framework , That is, your back propagation algorithm is defined by how your code works , Each iteration will be different .

Variable

autograd.variable Is the central class of this package . It's packed with a Tensor And support almost all definitions in Tensor The operating . Once you finish the calculation, you can call .backward() Make all your gradients automatically calculated

You can go through .data Feature access to any raw Tensor, About (w.r.t.with respect to) The gradient of this variable can be accumulated to .grad

h Another one is right autograd Implement very important classes Function
Variable and Function They are related to each other to build a cycle diagram , The graph encodes the entire history of the calculation . Every Variable There is one. .grad_fn characteristic , Point to creation Varaible Of Function ( except Varaibles Is created by the user , This is a grad_fn by None ).

If you want to calculate the derivative , It can be done to Variable call .backward(). If Variable It's a scalar , You don't need to be right backward() Indicate any parameters , But if Variable There are many elements , You have to specify a gradient parameter , This parameter is the tensor of the matching shape .

import torch 
from torch.autograd import Variable
x = Varaible(torch.ones(2, 2), requires_grad=True); #  Create a Variable
print(x)

y = x + 2
print(y)
#  because y Created as a result of an operation , So it has grad_fn
print(y.grad_dn)

z = y * y * 3
out = z.mean()
print(z, mean)

Gradient

Let us backprop, out.backward() Namely out.backward(torch.Tensor([1, 0]))
# It is quite given here x The initial value of the , Is to calculate the derivation formula and bring it in x Calculated x Value
out.backward();
print(x.grad)`
We should get a result that all elements are 4.5 Matrix ,
We call out by Variable”o”
$O = \frac{1}{4}\sum_{i}{Zi}$;
$Z_i = 3(x_i+2)^2$;
$Z_i|_{x_i = 1} = 27$;
$\frac{do}{dx} = \frac{3}{2}(x_i+2)$
$\frac{do}{dx_i}|_{x_i = 1} = \frac{9}{2} = 4.5$

You can use it. autograd Do a lot of crazy things

x = torch.randn(3)
x = Variable(x, requires_grad=True)

y = x * 2
while y.data.norm() < 1000:
    y = y * 2
print(y)

gradients = torch.FloatTensor([0.1, 1.0, 0.001])
y.backward(gradients)
# y  Yes  x  The result of the derivation is 1024
print(x.grad)
# result:
# 102.4000 
# 1024.0000
# 1.0240