Chapter 3 of hands on deep learning - (1) linear regression is realized from scratch_ Learning thinking and exercise answers

---

3.1 Linear regression

author github link ： github link

Learning notes

exercises

1. If we initialize the weight to zero , What's going to happen . Is the algorithm still valid ？
2. Suppose you are George · Simon · ohm , This paper attempts to establish a model for the relationship between voltage and current . Can you use automatic differentiation to learn the parameters of the model ?
3. You can be based on Planck's law Use spectral energy density to determine the temperature of an object ？
4. If you want to calculate the second derivative, what problems may you encounter ？ How would you solve these problems ？
5. Why is it squared_loss You need to use... In the function reshape function ？
6. Try using different learning rates , Observe how fast the value of the loss function decreases .
7. If the number of samples cannot be divided by the batch size ,data_iter What happens to the behavior of functions ？

Problem solving

1. If we initialize the weight to zero , What's going to happen . Is the algorithm still valid ？

Experiments show that the algorithm is still effective , It looks better
whole 0 Initialization is also a common choice , Compared with normal distribution initialization, it may move towards different local optima , The algorithm is still effective .

# w = torch.normal(0, 0.01, size=(2,1), requires_grad=True)
w = torch.zeros((2,1) ,requires_grad=True)

epoch 1, loss 0.036967
epoch 2, loss 0.000132
epoch 3, loss 0.000050

4. If you want to calculate the second derivative, what problems may you encounter ？ How would you solve these problems ？

The calculation formula of first-order derivative function cannot be obtained directly . resolvent ： Find the first derivative and save the calculation diagram .

Example ：$y=x^3+cosx,x=\frac{\pi }{2},\pi$, Find the first derivative and the second derivative respectively
Reference resources

import torch
import math
import numpy as np

x = torch.tensor([math.pi / 2, math.pi], requires_grad=True)
y = x ** 3 + torch.cos(x)

true_dy = 3 * x ** 2 - torch.sin(x)
true_d2y = 6 * x - torch.cos(x)

#  Find the first derivative , After saving the calculation diagram , To find the second derivative 
dy = torch.autograd.grad(y, x,
                         grad_outputs=torch.ones(x.shape),
                         create_graph=True,
                         retain_graph=True)  #  Keep the calculation diagram for calculating the second derivative 
#  After the tensor, add .detach().numpy() Only tensor values can be output 
print(" First derivative true value ：{} \n First derivative calculation value ：{}".format(true_dy.detach().numpy(), dy[0].detach().numpy()))

#  Find the second derivative . above dy The first element of is the first derivative 
d2y = torch.autograd.grad(dy, x,
                          grad_outputs=torch.ones(x.shape),
                          create_graph=False  #  No more calculation charts , Destroy the previous calculation diagram 
                          )
print("\n Second order conduction true value ：{} \n Second derivative calculation value ：{}".format(true_d2y.detach().numpy(), d2y[0].detach().numpy()))

5. Why is it squared_loss You need to use... In the function reshape function ？

$\hat{y}$ It's a column vector ,$y$ It's a row vector

6. Try using different learning rates , Observe how fast the value of the loss function decreases .

Try it on your own ：

• Low learning rate loss The decline is relatively slow
• Excessive learning rate loss Unable to converge

7. If the number of samples cannot be divided by the batch size ,data_iter What happens to the behavior of functions ？

The number of samples left at the end of execution cannot be divided , Will report a mistake