Introduction notes to pytorch deep learning (XII) neural network - nonlinear activation

Course notes , Course link
The learning notes are synchronously posted on my Personal website On , Welcome to check .

One 、 Introduction to common functions of nonlinear activation

The purpose of nonlinear activation is to introduce some nonlinear characteristics into our neural network .
Still open Official documents ：

The commonly used function is nn.ReLu:

1.1 ReLU

The corresponding function diagram is ：

Parameters ：inplace=True when , Will modify input Is the result of nonlinear activation ;inplace=False when , It doesn't change input ,input Still the original value .

Sample code ：

import torch
from torch import nn
from torch.nn import ReLU

input = torch.tensor([[1, -0.5],
                      [-1, 3]])
output = torch.reshape(input, (-1, 1, 2, 2))

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.relu1 = ReLU()

    def forward(self, input):
        output = self.relu1(input)
        return output

net1 = Net()
output = net1(input)
print(output)

Output ：

You can see that the input is Output is
[[1, -0.5], [[1, 0],
[-1, 3]], [0, 3]]
Input passed to ReLU function , Truncated ,-0.5 and -1 All less than 0, So the corresponding output is 0.

1.2 Sigmoid

The function diagram ：

Sample code ：

import torch
import torchvision
from torch import nn
from torch.nn import ReLU, Sigmoid
from torch.utils.data import DataLoader
from torch.utils.tensorboard import SummaryWriter

dataset = torchvision.datasets.CIFAR10("./dataset", train=False, download=True,
                                       transform=torchvision.transforms.ToTensor())
dataloader = DataLoader(dataset, batch_size=64)

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.Sigm = Sigmoid()

    def forward(self, input):
        output = self.Sigm(input)
        return output

net1 = Net()

writer = SummaryWriter("logs")
step = 0
for data in dataloader:
    imgs, targets = data
    writer.add_images("input", imgs, global_step=step)
    output = net1(imgs)
    writer.add_images("output", output, step)
    step += 1

writer.close()

use tensorboard View output ：

It can be seen that the main purpose of nonlinear activation is to add nonlinear characteristics to the network , In order to train a generalization model that meets the requirements .