Function of activation function

The activation function can introduce non-linear factors into the linear model , So as to solve the problem that linear model is difficult to solve .

Design a classification to separate the triangle and circle above , Take perceptron as an example , Consider a few situations ：

• Single layer perceptron
• Multilayer perceptron
• Single layer perceptron + Activation function
• Multilayer perceptron + Activation function

1 Single layer perceptron

The expression of single-layer perceptron is , It can draw a line , Divide the plane . For input features And characteristics , If , Prove to be a positive class ; If , Prove to be a negative class . We won't discuss it here In special circumstances . According to this, we can draw the coordinate map on the right .

No matter how the straight line obtained by the perceptron moves, it cannot separate the triangle from the circle .

2 Multilayer perceptron

The expression of multi-layer perceptron is shown in the formula on the right of the above figure , After the expression merges similar items , You can get , It can be found that no matter how it is combined , The final result is about the input linear equation, which can't deal with non-linear classification .

The straight line obtained by the multi-layer perceptron cannot separate the triangle from the circle no matter how it moves .

3 Single layer perceptron + Activation function

Add another one to the output of the sensor sigmoid Activation function , because sigmoid Is a non-linear function , So the output is obviously a nonlinear function , It is possible to solve the non-linear classification problem mentioned above .

Single layer perceptron + The curve obtained by activating the function may separate the triangle from the circle .

4 Multilayer perceptron + Activation function

Add the activation function to the output of the multi-layer perceptron , You can get the expression on the right . Because each layer is a non-linear output , The final output will also be a non-linear function , It is also possible for the non-linear classification problem mentioned above .

Multilayer perceptron + The curve obtained by activating the function may separate the triangle from the circle .

3、4 By constantly optimizing the loss function , Can learn to correctly classify the curves of triangles and circular points , As shown in the figure below 3 Curves .