Pytoch (II) -- activation function, loss function and its gradient

1. Activation function

1.1 Sigmoid / Logistic

$$\begin{gathered} \delta (x)=\frac{1}{1+e^{-x}} \\ {\delta }^{\prime }(x)=\delta (1-\delta ) \end{gathered}$$

import matplotlib.pyplot as plt
import torch.nn.functional as F
x = torch.linspace(-10,10,1000)
y = F.sigmoid(x)
plt.plot(x,y)
plt.show()

1.2 Tanh

$$\begin{gathered} tanh(x)=\frac{e^x-e^{-x}}{e^x+e^{-x}} \\ \frac{\partial tanh(x)}{\partial x}=1-tan{h}^2(x) \end{gathered}$$

import matplotlib.pyplot as plt
import torch.nn.functional as F
x = torch.linspace(-10,10,1000)
y = F.tanh(x)
plt.plot(x,y)
plt.show()

1.3 ReLU

$$f(x)=max(0,x)$$

import matplotlib.pyplot as plt
import torch.nn.functional as F
x = torch.linspace(-10,10,1000)
y = F.relu(x)
plt.plot(x,y)
plt.show()

1.4 Softmax

$$\begin{gathered} p_i=\frac{e^{a_i}}{{\sum }_{k=1}^N{e}^{a_k}} \\ \frac{\partial p_i}{\partial a_j}=\{\begin{array}{lc}{p}_i(1-p_j)& i=j\\ -p_i{p}_j& i\ne j\end{array} \end{gathered}$$

import torch.nn.functional as F
logits = torch.rand(10)
prob = F.softmax(logits,dim=0)
print(prob)

tensor([0.1024, 0.0617, 0.1133, 0.1544, 0.1184, 0.0735, 0.0590, 0.1036, 0.0861,
        0.1275])

2. Loss function

2.1 MSE

import torch.nn.functional as F
x = torch.rand(100,64)
w = torch.rand(64,1)
y = torch.rand(100,1)
mse = F.mse_loss(y,[email protected])
print(mse)

tensor(238.5115)

2.2 CorssEntorpy

import torch.nn.functional as F
x = torch.rand(100,64)
w = torch.rand(64,10)
y = torch.randint(0,9,[100])
entropy = F.cross_entropy([email protected],y)
print(entropy)

tensor(3.6413)

3. Derivation and back propagation

3.1 Derivation

• Tensor.requires_grad_()
• torch.autograd.grad()

import torch.nn.functional as F
import torch
x = torch.rand(100,64)
w = torch.rand(64,1)
y = torch.rand(100,1)
w.requires_grad_()
mse = F.mse_loss([email protected],y)
grads = torch.autograd.grad(mse,[w])
print(grads[0].shape)

torch.Size([64, 1])

3.2 Back propagation

• Tensor.backward()

import torch.nn.functional as F
import torch
x = torch.rand(100,64)
w = torch.rand(64,10)
w.requires_grad_()
y = torch.randint(0,9,[100,])
entropy = F.cross_entropy([email protected],y)
entropy.backward()
w.grad.shape

torch.Size([64, 10])

by CyrusMay 2022 06 28

life It's just In a moment
human It's just Between heaven and earth
—————— May day （ Because of you So I ）——————