CV learning notes - BP neural network training example (including detailed calculation process and formula derivation)

BP Examples of neural network training

1. BP neural network

About BP Neural network in my last blog 《CV Learning notes - Reasoning and training 》 Has been introduced in , I won't go into details here . Some of the things involved in this article are about BP The concept and basic knowledge of neural network are 《CV Learning notes - Reasoning and training 》 in , This article only deduces the process of examples .

BP The basic idea of Algorithm ：

• Input the training set data into the input layer of neural network , Through the hidden layer , Finally, it reaches the output layer and outputs the results , This is the former
  To the dissemination process .
• Due to the error between the output result of neural network and the actual result , Then calculate the error between the estimated value and the actual value , And the error
  Back propagation from the output layer to the hidden layer , Until it propagates to the input layer ;
• In the process of back propagation , Adjust the values of various parameters according to the error （ The weight of connected neurons ）, Make the total loss function minus
  Small .
• Iterate over the above three steps （ That is to train the data repeatedly ）, Until the stop criteria are met .

2. Training examples

1. Example design

The green node is the first input layer , Each node represents a neuron , among $i_1$、$i_2$ Represents the input value ,$b_1$ Is the offset value , The second layer contains $h_1$ and $h_2$ Two nodes , For the hidden layer ,$h_1$ and $h_2$ Input value of neuron ,$b_2$ Is the offset value of the hidden layer , The third layer is the output layer , Include $o_1$ and $o_2$,$w_1$~$w_8$ Is the weight between layers , Activate function using sigmoid function , The input value is $[i_1=0.05,i_2=0.10]$, The correct output value is $[o_1=0.01,o_2=0.99]$.

sigmoid Function is an activation function , In my last blog post 《CV Learning notes - Reasoning and training 》 Has been introduced in , No more details here .

2. Training process

1. Forward propagation

Input layer -> Hidden layer ：

According to the network structure diagram , Neuron $h_1$ Receive the previous layer $i_1$ and $i_2$ As input , Use this input $z_{h1}$ Express , Then there are
$$\begin{array}{rl}{z}_{h1}& =w_1\times i_1+w_2\times i_2+b_1\times 1\\ & =0.15\times 0.05+0.2\times 0.1+0.35\times 1\\ & =0.3775\end{array}$$
Because the activation function is sigmoid function , So neurons $h_1$ Output $a_{h1}$ by
$$a_{h1}=\frac{1}{1+e^{-z_{h1}}}=\frac{1}{1+e^{-0.3775}}=0.593269992$$
The same can be , Neuron $h_2$ Output $a_{h2}$ by
$$a_{h2}=0.596884378$$
Hidden layer -> Output layer ：

According to the network structure diagram , Neuron $o_1$ The input of $z_{o1}$ From the previous layer $h_1$ and $h_2$ Weighted sum result of , so
$$\begin{array}{rl}{z}_{o1}& =w_5\times a_{h1}+w_6\times a_{h2}+b_2\times 1\\ & =0.4\times 0.593269992+0.45\times 0.596884378+0.6\times 1\\ & =1.105905967\end{array}$$
Similarly, we can calculate $z_{o2}$

Because of the use of the Internet sigmoid The function is the activation function , that $o_1$ Output $a_{o1}$ by
$$\begin{array}{rl}{a}_{o1}& =\frac{1}{1+e^{-z_{o1}}}\\ & =\frac{1}{1+e^{-1.105905967}}\\ & =0.751365069\end{array}$$
Similarly, we can calculate $a_{o2}=0.772928465$

thus , The output value of a complete forward propagation process is $[0.751365069,0.772928465]$, And actual value $[0.01,0.99]$ The error is quite large , Back propagation of errors is required , Recalculate after updating the weight .

2. Back propagation

Calculate the loss function ：

The transfer error needs to be treated by the loss function , To estimate the appropriate transfer value for back-propagation and reasonably update the weight .
$$\begin{gathered} E_{total}=\sum \frac{1}{2}(target-output{)}^2 \\ {E}_{o1}=\frac{1}{2}(0.01-0.751365069{)}^2=0.274811083 \\ {E}_{o2}=\frac{1}{2}(0.99-0.772928465{)}^2=0.023560026 \\ {E}_{total}=E_{o1}+E_{o2}=0.298371109 \end{gathered}$$
Hidden layer -> Output layer weight update ：

With the weight parameter $w_5$ For example , Use the overall loss to $w_5$ After partial derivation, we can get $w_5$ Contribution to the overall loss , namely
$$\frac{\partial E_{total}}{\partial w_5}=\frac{\partial E_{total}}{\partial a_{o1}}\times \frac{\partial a_{o1}}{\partial z_{o1}}\times \frac{\partial z_{o1}}{\partial w_5}$$
$\frac{\partial E_{total}}{\partial E_{a_{o1}}}$： Since the total loss is caused by two outputs ($a_{o1}$ and $a_{o}2$) It works out , Therefore, the overall loss can be on $a_{o1}、a_{o2}$ Finding partial derivatives .

$\frac{\partial a_{o1}}{\partial z_{o1}}$： Due to output $a_{o1}$ It's input $z_{o1}$ adopt sigmoid Function is activated , so $a_{o1}$ It can be done to $z_{o1}$ Finding partial derivatives .

$\frac{\partial z_{o1}}{\partial w_5}$： because $z_{o1}$ It is from the previous network $h_1$ The output of is based on the weight $w_5$ Weighted sum results in , so $z_{o1}$ It can be done to $w_5$ Finding partial derivatives .

Clear the relationship between the above formulas ,$w_5$ Contributed to $z_{o1}$,$z_{o1}$ Contributed to $a_{o1}$, and $a_{o1}$ Contributed again $E_{total}$, All hierarchical relationships are unique branches , Therefore, it is directly disassembled into the solution of the above formula , This hierarchical relationship is the hidden layer described in the following chapters -> The weight update process of the hidden layer will be a little more complicated .

After the above derivation , It can be calculated as ：

$\frac{\partial E_{total}}{\partial a_{o1}}$ :
$$\begin{array}{rl}{E}_{total}& =\frac{1}{2}(targe{t}_{o1}-a_{o1}{)}^2+\frac{1}{2}(targe{t}_{o2}-a_{o2}{)}^2\\ \frac{\partial E_{total}}{\partial a_{o1}}& =2\times \frac{1}{2}(targe{t}_{o1}-a_{o1})\times (-1)\\ & =-(targe{t}_{o1}-a_{o1})\\ & =0.751365069-0.01\\ & =0.741365069\end{array}$$
$\frac{\partial a_{o1}}{\partial z_{o1}}$：
$$\begin{array}{rl}{a}_{o1}& =\frac{1}{1+e^{-z_{o1}}}\\ \frac{\partial a_{o1}}{\partial z_{o1}}& =a_{o1}\times (1-a_{o1})\\ & =0.751365069\times (1-0.751365069)\\ & =0.186815602\end{array}$$
$\frac{\partial z_{o1}}{\partial w_5}$：
$$\begin{array}{rl}{z}_{o1}& =w_5\times a_{h1}+w_6\times a_{h2}+b_2\times 1\\ \frac{\partial z_{o1}}{\partial w_5}& =a_{h1}\\ & =0.593269992\end{array}$$
From the above three results , Available ：
$$\frac{\partial E_{total}}{\partial w_5}=0.741365069\times 0.186815602\times 0.593269992=0.082167041$$
If we remove specific values from the above steps , Abstract it out

Then we get
$$\begin{gathered} \frac{\partial E_{total}}{\partial w_5}=-(targe{t}_{o1}-a_{o1})\times a_{o1}\times (1-a_{o1})\times a_{h1} \\ \frac{\partial E}{\partial w_{jk}}=-(t_k-o_k)\cdot sigmoid(\sum _j{w}_{jk}\cdot o_j)(I-sigmoid(\sum _j{w}_{jk}\cdot o_j))\cdot o_j \end{gathered}$$

The formula in the second line was mentioned in my last blog , Now the derivation is made .

For the convenience of expression , use ${\delta }_{o1}$ To represent the error of the output layer ：
$$\begin{gathered} {\delta }_{o1}=\frac{\partial E_{total}}{\partial a_{o1}}\times \frac{\partial a_{o1}}{\partial z_{o1}}=\frac{\partial E_{total}}{\partial z_{o1}} \\ {\delta }_{o1}=-(targe{t}_{o1}-a_{o1})\times a_{o1}\times (1-a_{o1}) \end{gathered}$$
Therefore, the overall loss for $w_5$ The partial derivative of can be simply expressed as
$$\frac{\partial E_{total}}{\partial w_5}={\delta }_{o1}\times a_{h1}$$
be $w_5$ The weight of is updated to ：
$$\begin{array}{rl}{w}_5^{+}& =w_5-\eta \times \frac{\partial E_{total}}{\partial w_5}\\ & =0.4-0.5\times 0.082167041\\ & =0.35891648\end{array}$$

$\eta$ For learning rate , In my last blog post 《CV Learning notes - Reasoning and training 》 Introduced in , I won't repeat .

Empathy , renewable $w_6,w_7,w_8$：
$$\begin{gathered} w_6^{+}=0.408666186 \\ {w}_7^{+}=0.511301270 \\ {w}_8^{+}=0.561370121 \end{gathered}$$
Hidden layer -> Update the weight of hidden layer ：

Their ideas are roughly the same , But the difference is $h_1$ Output $a_{h1}$ Yes $E_{o1}、E_{o2}$ All have contributed , Therefore, the loss is generally right $a_{h1}$ When seeking partial derivative , According to the criterion of total differentiation , Be divided into pairs $E_{o1}、E_{o2}$ Yes $a_{h1}$ Partial derivative of , namely
$$\begin{gathered} \frac{\partial E_{total}}{\partial w_1}=\frac{\partial E_{total}}{\partial a_{h1}}\times \frac{\partial a_{h1}}{\partial z_{h1}}\times \frac{\partial z_{h1}}{\partial w_1} \\ Itsin：\frac{\partial E_{total}}{\partial a_{h1}}=\frac{\partial E_{o1}}{\partial a_{h1}}\times \frac{\partial E_{o2}}{\partial a_{h1}} \end{gathered}$$

From the above derivation , Calculated ：

$\frac{\partial E_{total}}{\partial a_{h1}}$：
$$\frac{\partial E_{total}}{\partial a_{h1}}=\frac{\partial E_{o1}}{\partial a_{h1}}\times \frac{\partial E_{o2}}{\partial a_{h1}}$$
$\frac{\partial E_{o1}}{\partial a_{h1}}$：
$$\begin{array}{rl}\frac{\partial E_{o1}}{\partial a_{h1}}& =\frac{\partial E_{o1}}{\partial a_{o1}}\times \frac{\partial a_{o1}}{\partial z_{o1}}\times \frac{\partial z_{o1}}{\partial a_{h1}}\\ & =0.741365069\times 0.186815602\times 0.4\\ & =0.055399425\end{array}$$

The same can be ：
$$\frac{\partial E_{o2}}{\partial a_{h1}}=-0.019049119$$
The two add up to ：
$$\begin{array}{rl}\frac{\partial E_{total}}{\partial a_{h1}}& =\frac{\partial E_{o1}}{\partial a_{h1}}\times \frac{\partial E_{o2}}{\partial a_{h1}}\\ & =0.055399435-0.019049119\\ & =0.036350306\end{array}$$
$\frac{\partial a_{h1}}{\partial z_{h1}}$：
$$\begin{array}{rl}\frac{\partial a_{h1}}{\partial z_{h1}}& =a_{h1}\times (1-a_{h1})\\ & =0.593269992\times (1-0.593269992)\\ & =0.2413007086\end{array}$$
$\frac{\partial z_{h1}}{\partial w_1}$：
$$\frac{\partial z_{h1}}{\partial w_1}=i_1=0.05$$
final result ：
$$\frac{\partial E_{total}}{\partial w_1}=0.036350306\times 0.2413007086\times 0.05=0.000438568$$
The simplified method in the previous section , use ${\delta }_{h1}$ Represents hidden layer cells $h_1$ The error of the ：
$$\begin{array}{rl}\frac{\partial E_{total}}{\partial w_1}& =(\sum _i\frac{\partial E_{total}}{\partial a_i}\times \frac{\partial a_i}{\partial z_i}\times \frac{\partial z_i}{\partial a_{h1}})\times \frac{\partial a_{h1}}{\partial z_{h1}}\times \frac{\partial z_{h1}}{\partial w_1}\\ & =(\sum _i{\delta }_i\times w_{hi})\times a_{h1}\times (1-a_{h1})\times i_1\\ & ={\delta }_{h_1}\times i_1\end{array}$$
$w_1$ The weight of is updated to ：
$$w_1^{+}=w_1-\eta \times \frac{\partial E_{total}}{\partial w_1}=0.15-0.5\times 0.000438568=0.149780716$$
Empathy , to update $w_2,w_3,w_4$：
$$\begin{gathered} w_2^{+}=0.19956143 \\ {w}_3^{+}=0.24975114 \\ {w}_4^{+}=0.29950229 \end{gathered}$$
thus , The process of a back propagation ends .

This is how the training process iterates , Error after forward propagation , Update weights in back propagation , Then forward propagation , This goes on over and over again , After the first iteration of this example, the total error is from 0.298371109 Down to 0.291027924, In iteration 10000 Next time , The total error is reduced to 0.000035085. Output is [0.015912196,0.984065734]

---

Personal study notes , Just learn to communicate , Reprint please indicate the source ！