Some derivation formulas for machine learning backpropagation

1.预备知识

The understanding of matrix derivation can refer to the derivative that we are familiar with in high school,In high school we were all taking scalar derivatives,Scalar can also be regarded as a special kind1*1的矩阵.This article is mainly to document the process of backpropagation in machine learning,So don't do too much analysis on matrix derivation（In fact, neither will I,只会简单的）.

Only a matrix derivation situation that needs to be used in the back propagation process is given here：
$$\begin{array}{rl}\frac{\partial \left({\mathbf{a}}^T\mathbf{x}\right)}{\partial \mathbf{x}}& =\frac{\partial \left(a_1{x}_1+a_2{x}_2+\cdots +a_n{x}_n\right)}{\partial \mathbf{x}}\\ & =\left[\begin{array}{l}\frac{\partial \left(a_1{x}_1+a_2{x}_2+\cdots +a_n{x}_n\right)}{\partial x_1}\\ \frac{\partial \left(a_1{x}_1+a_2{x}_2+\cdots +a_n{x}_n\right)}{\partial x_2}\\ ⋮\\ \frac{\partial \left(a_1{x}_1+a_2{x}_2+\cdots +a_n{x}_n\right)}{\partial x_n}\end{array}\right]\\ & =\left[\begin{array}{l}{a}_1\\ a_2\\ ⋮\\ a_n\end{array}\right]\\ & =\mathbf{a}\end{array}$$
Once we understand this, we can start~

2.反向传播

We start propagating backwards：

隐藏层第2层：
$$激活函数:d{a}^{[2]}=\frac{\partial L}{\partial a^{[2]}}$$

$$d{z}^{[2]}=\frac{\partial L}{\partial z^{[2]}}=\frac{\partial L}{\partial a^{[2]}}\cdot \frac{\partial a^{[2]}}{\partial z^{[2]}}=d{a}^{[2]}\cdot g^{[2]’}(z^{[2]})$$

$$\begin{gathered} d{W}^{[2]}=\frac{\partial L}{\partial W^{[2]}}=\frac{\partial L}{\partial z^{[2]}}\cdot \frac{\partial z^{[2]}}{\partial W^{[2]}}=d{z}^{[2]}\cdot a^{[1]T} \\ \Rightarrow W^{[2]}-=\alpha \cdot d{W}^{[2]} \end{gathered}$$

$$\begin{gathered} d{b}^{[2]}=\frac{\partial L}{\partial b^{[2]}}=\frac{\partial L}{\partial z^{[2]}}\cdot \frac{\partial z^{[2]}}{\partial b^{[2]}}=d{z}^{[2]} \\ \Rightarrow b^{[2]}-=\alpha \cdot d{b}^{[2]} \end{gathered}$$

隐藏层第1层：
$$激活函数:d{a}^{[1]}=\frac{\partial L}{\partial a^{[1]}}=\frac{\partial L}{\partial z^{[2]}}\cdot \frac{\partial z^{[2]}}{\partial a^{[1]}}=W^{[2]T}\cdot d{z}^{[2]}$$
说实话,I don't understand the calculation result of this step：

$\frac{\partial L}{\partial z^{[2]}}$ 是 $d{z}^{[2]}$,$\frac{\partial z^{[2]}}{\partial a^{[1]}}$ 是 $W^{[2]T}$,Why is the result of multiplying $W^{[2]T}\cdot d{z}^{[2]}$,而不是 $d{z}^{[2]}\cdot W^{[2]T}$.

$$d{z}^{[1]}=\frac{\partial L}{\partial z^{[1]}}=\frac{\partial L}{\partial a^{[1]}}\cdot \frac{\partial a^{[1]}}{\partial z^{[1]}}=d{a}^{[1]}\cdot g^{[1]’}(z^{[1]})$$

$$\begin{gathered} d{W}^{[1]}=\frac{\partial L}{\partial W^{[1]}}=\frac{\partial L}{\partial z^{[1]}}\cdot \frac{\partial z^{[1]}}{\partial W^{[1]}}=d{z}^{[1]}\cdot a^{[0]T} \\ \Rightarrow W^{[1]}-=\alpha \cdot d{W}^{[1]} \end{gathered}$$

$$\begin{gathered} d{b}^{[1]}=\frac{\partial L}{\partial b^{[1]}}=\frac{\partial L}{\partial z^{[1]}}\cdot \frac{\partial z^{[1]}}{\partial b^{[1]}}=d{z}^{[1]} \\ \Rightarrow b^{[1]}-=\alpha \cdot d{b}^{[1]} \end{gathered}$$

3.总结

第l层：
$$激活函数:d{a}^{[l]}=\frac{\partial L}{\partial a^{[l]}}=\frac{\partial L}{\partial z^{[l+1]}}\cdot \frac{\partial z^{[l+1]}}{\partial a^{[l]}}=W^{[l+1]T}\cdot d{z}^{[l+1]}$$

$$\begin{gathered} d{z}^{[l]}=\frac{\partial L}{\partial z^{[l]}}=\frac{\partial L}{\partial a^{[l]}}\cdot \frac{\partial a^{[l]}}{\partial z^{[l]}}=d{a}^{[l]}\cdot g^{[l]’}(z^{[l]}) \\ \Rightarrow d{z}^{[l]}=W^{[l+1]T}d{z}^{[l+1]}\cdot g^{[l]’}(z^{[l]}) \end{gathered}$$

$$\begin{gathered} d{W}^{[l]}=\frac{\partial L}{\partial W^{[l]}}=\frac{\partial L}{\partial z^{[l]}}\cdot \frac{\partial z^{[l]}}{\partial W^{[l]}}=d{z}^{[l]}\cdot a^{[l-1]T} \\ \Rightarrow W^{[l]}-=\alpha \cdot d{W}^{[l]} \end{gathered}$$

$$\begin{gathered} d{b}^{[l]}=\frac{\partial L}{\partial b^{[l]}}=\frac{\partial L}{\partial z^{[l]}}\cdot \frac{\partial z^{[l]}}{\partial b^{[l]}}=d{z}^{[l]} \\ \Rightarrow b^{[l]}-=\alpha \cdot d{b}^{[l]} \end{gathered}$$