Neural network - nonlinear activation

summary ：

Nonlinear activation is mainly to introduce some nonlinear characteristics into our neural network , Two commonly used nonlinear activation are Sigmoid and RELU.

RELU

Sample code ：

import torch
from torch import nn
from torch.nn import ReLU

input = torch.tensor([[1,-0.5],
                      [-1,3]])

input = torch.reshape(input,(-1,1,2,2))
print(input.shape)

#  Building neural networks 
class Booze(nn.Module):

    def __init__(self):
        super(Booze, self).__init__()
        # inplace The meaning of the parameter is whether to replace the original variable value with the processed result value   if inplace=True Replace the original variable , The value of the original variable changes , Become the result value after processing , if inplace=False Then the original variable will not be replaced , The result needs a new variable to receive 
        #  Usually , It is suggested that inplace Pass in False, This ensures that the original data is not lost 
        self.relu1 = ReLU()
	#  rewrite forword Method 
    def forward(self,input):
        output = self.relu1(input)
        return output

obj = Booze()
output = obj(input)
print(output)

Code run results ：

about ReLU Use of interfaces , This interface has only one parameter that needs to be passed , That's it inplace Parameters .

torch.nn.ReLU(inplace=False)

inplace The meaning of the parameter is whether to replace the original variable value with the processed result value if inplace=True Replace the original variable , The value of the original variable changes , Become the result value after processing , if inplace=False Then the original variable will not be replaced , The result needs a new variable to receive .
Usually , It is suggested that inplace Pass in False, This ensures that the original data is not lost .

The picture above is ReLU Activate the image of the function , Explanation of the above figure ： Less than zero for truncation , If it is greater than zero, the original value will be output .

Sigmoid

Sample code ：

import torch
import torchvision
from torch import nn
from torch.nn import ReLU, Sigmoid
from torch.utils.data import DataLoader
from torch.utils.tensorboard import SummaryWriter


dataset = torchvision.datasets.CIFAR10("./CIFAR10",train=False,transform=torchvision.transforms.ToTensor(),download=True)
dataloader = DataLoader(dataset,batch_size=64)

#  Building neural networks 
class Booze(nn.Module):

    def __init__(self):
        super(Booze, self).__init__()
        # inplace The meaning of the parameter is whether to replace the original variable value with the processed result value   if inplace=True Replace the original variable , The value of the original variable changes , Become the result value after processing , if inplace=False Then the original variable will not be replaced , The result needs a new variable to receive 
        #  Usually , It is suggested that inplace Pass in False, This ensures that the original data is not lost 
        self.relu1 = ReLU()
        self.sigmoid1 = Sigmoid()

    #  rewrite forward Method 
    def forward(self,input):
        # output = self.relu1(input)
        output = self.sigmoid1(input)
        return output

obj = Booze()

#  Use tensorboard Visualizing 
writer = SummaryWriter('logs')
step = 0

for data in dataloader:
    imgs,targets = data
    writer.add_images("input",imgs,step)
    #  Use neural network to process pictures 
    output = obj(imgs)
    writer.add_images("output",output,step)

    step+=1

writer.close()

Sigmoid and RELU The use method of is similar, so I won't repeat it here .
The above code results in tensorboard The visualization in is shown in the following figure ：

Nonlinear activation sigmoid The effect before and after function processing is still quite obvious .

summary

The main purpose of nonlinear transformation is to introduce some nonlinear features into our network . Because there are many nonlinearity , You can train a model that conforms to various curves or characteristics , If everyone is straight , The generalization ability of the model is not good enough .