Watermelon book -- Chapter 5 neural network

Individuality signature ： The most important part of the whole building is the foundation , The foundation is unstable , The earth trembled and the mountains swayed .

                  And to learn technology, we should lay a solid foundation , Pay attention to me , Take you to firm the foundation of the neighborhood of each plate .

Blog home page ： Qigui's blog

It's not easy to create , Don't forget to hit three in a row when you pass by ！！！
Focus on the author , Not only lucky , The future is more promising ！！！

Triple attack( Three strikes in a row ):Comment,Like and Collect--->Attention

Artificial neural network （ANN）： Simulate the structure and function of human brain nervous system , An artificial network system composed of a large number of simple processing units and widely connected .

One 、 Neuron

A neuron usually has multiple dendrites . It is mainly used to receive incoming information , There is only one axon . Axons can transmit information to many other neurons . Axon terminals connect with dendrites , To send a signal , This connection corresponds to a weight ; The value of the weight is called the weight , This is something that needs training . namely Each connecting line corresponds to a different weight .

Neuron model Is a containing input , Models of output and computing functions . The input can be likened to the dendrites of neurons , The output can be compared to the axon of a neuron , The calculation can be compared to the nucleus . The most basic component of neural network is neuron model .

Again, connection is the most important thing in neurons , Each connection has a weight .

A training algorithm of neural network is to adjust the value of weight to the best , So that the prediction effect of the whole network is the best .

Which has been used until now is M-P Neuron model . In this model , Neurons receive information from n The input signals from these other neurons , These input signals are transmitted through the connection of weights , The total input received by the neuron is compared with the threshold of the neuron , And then through Activation function Processing to produce the output of neurons . 

Next, it's about Calculation of neurons , Input It's been through Three step mathematical operation ：

1. First enter multiply by weight （weight）：x1-->x1 * w1;x2-->x2 * w2

2. Sum up ：(x1 * w1) + (x2 * w2) + b

3. After the activation function processing, the output ：y = f((x1 * w1) + (x2 *w2) +b)

Activation function ：

Nonlinear function is introduced into neural network as activation function , It is no longer a linear combination of inputs , But almost any function .

The function of activation ： Convert unlimited input into predictable output , The commonly used activation function is sigmoid function .

sigmoid Function squeezes input values that may vary over a wide range to （0,1） Output value range , So it is sometimes called ” Squeeze function “.

import matplotlib.pyplot as plt
import numpy as np


def sigmoid(x):
    #  Go straight back to sigmoid function 
    return 1 / (1 + np.exp(-x))


def plot_sigmoid():
    # param: The starting point , End , spacing 
    x = np.arange(-8, 8, 0.1)
    y = sigmoid(x)
    plt.plot(x, y)
    plt.show()


if __name__ == '__main__':
    plot_sigmoid()

Two 、 neural network

Multilayer neural network , Not at all The more levels, the better .

How to build neural networks ：

1. Building a neural network is to connect multiple neurons .

2. This neural network has 2 Inputs 、 A contain 2 A hidden layer of neurons （h1 and h2）、 contain 1 The output layer of neurons o1.

3. The hidden layer is the part sandwiched between the input layer and the output layer , A neural network can have multiple hidden layers .

  3、 ... and 、 perceptron ( Reference resources 《 Statistical learning method 》) And multi tier Networks

The perceptron consists of two layers of neurons . Input layer After receiving the external input signal, it is transmitted to the output layer , Output layer yes M-P Neuron , Also known as ” Threshold logical unit “.

Perceptron is a linear classification model of binary classification problem . Single layer perceptron can only deal with linear problems , Can't handle nonlinear problems ！！！ The perceptron has only output layer neurons to process the activation function , That is, only one layer of functional neurons . To solve nonlinear problems , We need to consider using multi-layer functional neurons . The layer of neurons between the output layer and the input layer is called Hidden layer or hidden layer , Hidden layer and output layer neurons are functional neurons with activation function .

Each layer of neurons is fully interconnected with the next layer of neurons , There is no same layer connection between neurons , There is no cross layer connection , Such a neural network structure is usually called ” Multilayer feedforward neural network “.

Input layer neurons only receive input without performing functions beyond , The hidden layer and the output layer contain functional neurons , It's called ” Two layer network “; Just include the hidden layer , It can be called ” Multi layer network “.

Four 、 Error back propagation algorithm （BP Algorithm ）

BP It's an iterative learning algorithm , In each iteration, the generalized perceptron learning rules are used to update the parameters .

Common activation function selection ：sigmoid function 、tanh function 、ReLU function 、Leaky ReLU function .

Algorithm flow ：

1. Provide input samples to input layer nerves element

2. Layer by layer Signal forward To the hidden layer 、 Output layer , Produce the result of the output layer

3. Calculate the output layer error

4. take Error back propagation To the hidden nerve element

5. According to the connection weights of hidden layer neurons and Adjust the threshold

6. The above process is carried out circularly , Until it reaches Until certain stop conditions  

Before training neural network, we need to have a standard to define whether it is good or not , In order to improve , This is it. Loss function , For example, oral pragmatic mean square error is used to define loss .

Mean square error Is the average of all data variances . Define it as the loss function , The better the forecast , The lower the loss , Training neural network is to minimize the loss . therefore ,BP The goal of the algorithm is to minimize the cumulative error .

import numpy as np

def mse_loss(y_true, y_pre):
    # y_true  and  y_pre  It's the same length np Array  
    return ((y_true - y_pre) ** 2).mean()

Test code ：

y_true = np.array([1, 0, 0, 1])
y_pre = np.array([0, 0, 0, 0])

print(mse_loss(y_true, y_pre))  # 0.5

But because of BP The powerful ability of has often encountered fitting , Reduce the training error , But the test error has increased . So there are two ways relieve BP Over fitting of the algorithm ：1.‘ Stop early ’;2.‘ Regularization ’. 

Chapter six - Support vector machine (SVM)
http://t.csdn.cn/q6o2Fhttp://t.csdn.cn/q6o2F Chapter four - Decision tree http://t.csdn.cn/3Tme3http://t.csdn.cn/3Tme3 The third chapter - Linear model http://t.csdn.cn/4S6Y6http://t.csdn.cn/4S6Y6