1.5 nn. Module neural network (III)

1. Optimizer

1.1 After the gradient of all parameters is calculated by back propagation , We also need to use optimization methods to update the weights and parameters of the network . for example , Random gradient descent method （SGD） The update strategy is as follows ：

weight= weight-learning_rate*gradient

Manual implementation is as follows ：

learning_rate=0.01
for f in net.parameters():
    f.data.sub_(f.grad_data*learning_rate)#inplace Subtraction 

1.2. torch.optim It realizes most optimization methods in deep learning , for example ：RMSProp、Adam、SGD etc. , Easier to use , Therefore, there is usually no need to write the above code manually .

import torch.optim as optim
optimizer = optim.SGD(net.parameters(), lr=0.01)
# In the training process, first clear the gradient （ And net.zero_grad() The effect is the same ）
optimizer.zero_grad()
# Calculate the loss 
output=net(input)
criterion = nn.MSELoss()
loss = criterion(output,target)
# Back propagation 
loss.backward()
# Update parameters 
optimizer.step()

2. Data loading and preprocessing

torchvision The commonly used image data loading function is realized , for example Imagenet、CIFAR10、MNIST etc. , And common data conversion operations , Greatly facilitates data loading .