ML16 neural network (2)

neural network

The overall architecture

The middle phase is equivalent to the weight , The line in the first column , The first three inputs , The last four outputs , That's the weight W by 34 Matrix , Line in the second column , Input is 4, The output is 4, therefore W The weight matrix is 44 Of , The third column is the same . The result of hidden layer one is xw₁, And then after the second floor 【xw₁】*w₂, Through the third layer, the output result is obtained .
however , Why should there be multiple layers W Weight , Can't one ？

The following explains why there should be multiple layers W The weight , Because there is only one W So it's a linear equation , In that case, classification can only be divided into linear , It is not conducive to the accuracy of classification results . Add multiple weights W after , Become a nonlinear equation ,max Is the activation function , Activation functions can be added to each layer .

Activation function

sigmoid function

At first, neural network is to take sigmoid When activating function , But because when x Greater than 10 Or less than -10 When , Values are very small , The gradient vanishes , When the number of layers is very deep, it will lead to no update （ because W Is too small ）, It doesn't work much now .

ReLu function

Replaced the sigmoid function .

Demo example

Neural network demonstration example ：http://cs.stanford.edu/people/karpathy/convnetjs/demo/classify2d.html
Change the number of neurons , Will make the classification effect change , The more classification, the better . But it will cause over fitting .

Solve over fitting

λ=0.1 Time has strong generalization ability . stay λ=0.001 when , Those protruding red pointed parts , Although the red dots are well divided , However, the probability of green in this position is relatively high , When a green dot appears in this place, it is easy to judge as a red dot , This is over fitting .

The whole process of neural network .

Data preprocessing

Data processing

chart 2 Make the mean 0; chart 3 Reduce variance .

Weight initialization

Be careful ：w Cannot initialize with mean or zero .B The initialization of can be 0.

drop-out

Full connection , It is easy to cause over fitting .
Use drop-out Method , Choose no neurons at random each time .