Pytorch - autograd automatic differentiation

The paper ：Automatic Differentiation in Machine Learning: a Survey, Automatic differentiation

Reference resources PyTorch：AUTOMATIC DIFFERENTIATION WITH TORCH.AUTOGRAD

Loss It's scalar , Jacobian vector product ,JVP,

primitive operation( Primitive operation )：

torch.autograd(), Calculate the gradient

import torch

x = torch.ones(5)  # input tensor,  Input vector 
print(f"x: {x}")
y = torch.zeros(3)  # expected output,  label 
print(f"y: {y}")
w = torch.randn(5, 3, requires_grad=True)  #  Turn on automatic differentiation 
print(f"w: {w}")
b = torch.randn(3, requires_grad=True)
print(f"b: {b}")
z = torch.matmul(x, w)+b

loss = torch.nn.functional.binary_cross_entropy_with_logits(z, y)

Use gradient back propagation algorithm ,back propagation

backward() yes Tensor Class method ,loss Is a scalar direct call backward(),loss If it's a tensor , be backward() Need to pass in tensor

print(f"Gradient function for z = {
      z.grad_fn}")
print(f"Gradient function for loss = {
      loss.grad_fn}")

loss.backward()
print(w.grad)
print(b.grad)

retain_graph=True, Keep the diagram , If not reserved , The first 2 This call will report an error ：RuntimeError: Trying to backward through the graph a second time

torch.no_grad() Turn off automatic derivation ：

z = torch.matmul(x, w)+b
print(z.requires_grad)

with torch.no_grad():
    z = torch.matmul(x, w)+b
print(z.requires_grad)

z = z.detach() after ,z.requires_grad yes False

z = z.detach()
z.requires_grad

DAG：directed acyclic graph, Directed acyclic graph

tensor loss, Input torch.ones_like(inp), Back propagation

retain_graph Keep the diagram , Continuous backward()

Gradient set 0,np.grad.zero_()

inp = torch.eye(5, requires_grad=True)
out = (inp+1).pow(2)
print(out)
out.backward(torch.ones_like(inp), retain_graph=True)
print(f"First call\n{
      inp.grad}")
out.backward(torch.ones_like(inp), retain_graph=True)
print(f"\nSecond call\n{
      inp.grad}")
inp.grad.zero_()
out.backward(torch.ones_like(inp), retain_graph=True)
print(f"\nCall after zeroing gradients\n{
      inp.grad}")