A detailed explanation of vector derivative and matrix derivative

Vector derivative and matrix derivative are the mathematical basis of machine learning , Read this article carefully , I believe you will have a lot to gain ~

$carryAndtowardsThe\; amountwhen,ifnothing,Especiallysaybright,IPeopleSilentrecognizebyColumntowardsThe\; amount$

One 、 Numerator layout and denominator layout

We know , Scalar （Scalar）、 vector （Vector） And matrices （Matrix） The three of them meet the following relationship ：

$$markThe\; amount\subset towardsThe\; amount\subset Momentfront$$

That is, a vector can be understood as a special matrix （ The number of columns is $1$ Matrix ）, Scalar can be understood as a special vector （ Dimension for $1$ Vector ）, It can also be understood as a $1\times 1$ Matrix , So the vector derivative and matrix derivative we are discussing today can be collectively referred to as “ Matrix derivative ”.

There are six common matrix derivatives ：

Scalar derivative of scalar is familiar to everyone （$f^{\prime }(x)$ It's a typical example ）, We won't discuss it here . In fact, we can also discuss the derivative between matrix and vector , Derivative between matrices , That is, where the table is empty , But because the results of these derivatives involve dimensions greater than $2$ Tensor （tensor）, We can no longer express in the form of matrix , Therefore, we will not discuss .

Next, we will focus on the remaining five matrix derivatives , namely ：

• Vector derivative of scalar
• Scalar derivative of vector
• Vector to vector derivation
• Matrix derivative of scalar
• Scalar derivative of matrix

Suppose we have $\mathbf{x}=(x_1,\cdots ,x_n{)}^{\mathrm{T}}$ and $\mathbf{y}=(y_1,\cdots ,y_m{)}^{\mathrm{T}}$ Two vectors , be $\frac{\partial \mathbf{y}}{\partial \mathbf{x}}$ share $mn$ Weight ：

$$\frac{\partial y_i}{\partial x_j},i=1,\cdots ,mj=1,\cdots ,n$$

How should we arrange this $mn$ How many components ？ That's what we need Molecular arrangement （Numerator Layout） and Denominator layout （Denominator Layout） 了 . The so-called layout , It is nothing more than an arrangement of the above results , If the arrangement is not specified , It is likely to cause errors in the process of mathematical operations （ For example, due to the dimension of the matrix, it cannot be multiplied ）.

When talking about vector derivatives , We have two very important premises ：

① Molecules and denominators Is a vector , And one of them is Row vector , The other is Column vector
② One of the numerator and denominator is Scalar , The other is That's ok / Column vector

When ① or ② When satisfied , Our next discussion is meaningful .

We see first ①：

• If denominator is column vector , Molecules are row vectors , It is called denominator layout
• If the molecule is a column vector , The denominator is the row vector , It is called molecular layout

In a word, it is ： Who is the column vector is what layout .

about ②, We can still use “ Who is the column vector is what layout ” To judge , But if the numerator and denominator are not column vectors , How to judge ？

This situation is also summarized in one sentence ： Who is scalar is what layout .

We can summarize these discussions in the following table ：

about Matrix derivative , Things are a little different ：

Besides , We also have the following Important equation ：

$$branchSonclothgameOfjunctionfruit=branchmotherclothgameOfjunction{fruit}^{\mathrm{T}},branchmotherclothgameOfjunctionfruit=branchSonclothgameOfjunction{fruit}^{\mathrm{T}}$$

All the above results can be summarized into the following three figures ：

More intuitive expression ：

Our next discussion will be based on Molecular arrangement .

Two 、 Vector derivative

2.1 Vector derivative of scalar

Some rules for the derivation of vectors from scalars ：

2.2 Scalar derivative of vector

Some rules of scalar derivation from vector ：

Some of scalar derivatives of vectors Important conclusions ：

$$\frac{\partial a}{\partial \mathbf{x}}=0^{\mathrm{T}}\text{\hspace{1em}(2.2.A)}$$

$$\frac{\partial {\mathbf{a}}^{\mathrm{T}}\mathbf{x}}{\partial \mathbf{x}}=\frac{\partial {\mathbf{x}}^{\mathrm{T}}\mathbf{a}}{\partial \mathbf{x}}={\mathbf{a}}^{\mathrm{T}}\text{\hspace{1em}(2.2.B)}$$

$$\frac{\partial {\mathbf{x}}^{\mathrm{T}}\mathbf{x}}{\partial \mathbf{x}}=2{\mathbf{x}}^{\mathrm{T}}\text{\hspace{1em}(2.2.C)}$$

$$\frac{\partial {\mathbf{x}}^{\mathrm{T}}\mathbf{A}\mathbf{x}}{\partial \mathbf{x}}={\mathbf{x}}^{\mathrm{T}}(\mathbf{A}+{\mathbf{A}}^{\mathrm{T}})\text{\hspace{1em}(2.2.D)}$$

2.3 Vector to vector derivation

Some rules of vector to vector derivation ：

Some of the derivatives of vectors Important conclusions ：

$$\frac{\partial \mathbf{a}}{\partial \mathbf{x}}=\mathbf{O}\text{\hspace{1em}(2.3.A)}$$

$$\frac{\partial \mathbf{x}}{\partial \mathbf{x}}=\mathbf{I}\text{\hspace{1em}(2.3.B)}$$

$$\frac{\partial \mathbf{A}\mathbf{x}}{\partial \mathbf{x}}=\mathbf{A}\text{\hspace{1em}(2.3.C)}$$

3、 ... and 、 Matrix derivative

3.1 Matrix derivative of scalar

Some rules of matrix deriving scalar ：

3.2 Scalar derivative of matrix

Some rules of scalar matrix derivation ：

Some of scalar derivation of matrix Important conclusions ：

$$\frac{\partial a}{\partial \mathbf{X}}=\mathbf{O}\text{\hspace{1em}(3.2.A)}$$

$$\frac{\partial {\mathbf{a}}^{\mathrm{T}}\mathbf{X}\mathbf{b}}{\partial \mathbf{X}}={\mathbf{b}\mathbf{a}}^{\mathrm{T}}\text{\hspace{1em}(3.2.B)}$$

$$\frac{\partial {\mathbf{a}}^{\mathrm{T}}{\mathbf{X}}^{\mathrm{T}}\mathbf{b}}{\partial \mathbf{X}}={\mathbf{a}\mathbf{b}}^{\mathrm{T}}\text{\hspace{1em}(3.2.C)}$$

Besides , We often encounter the derivation of trace pair matrix , Relevant conclusions are as follows ：

$$\frac{\partial \mathrm{t}\mathrm{r}(\mathbf{X})}{\partial \mathbf{X}}=\mathbf{I}\text{\hspace{1em}(3.2.D)}$$

$$\frac{\partial \mathrm{t}\mathrm{r}({\mathbf{X}}^k)}{\partial \mathbf{X}}=k{\mathbf{X}}^{k-1}\text{\hspace{1em}(3.2.E)}$$

$$\frac{\partial \mathrm{t}\mathrm{r}(\mathbf{A}\mathbf{X})}{\partial \mathbf{X}}=\frac{\partial \mathrm{t}\mathrm{r}(\mathbf{X}\mathbf{A})}{\partial \mathbf{X}}=\mathbf{A}\text{\hspace{1em}(3.2.F)}$$

$$\frac{\partial \mathrm{t}\mathrm{r}({\mathbf{A}\mathbf{X}}^{\mathrm{T}})}{\partial \mathbf{X}}=\frac{\partial \mathrm{t}\mathrm{r}({\mathbf{X}}^{\mathrm{T}}\mathbf{A})}{\partial \mathbf{X}}={\mathbf{A}}^{\mathrm{T}}\text{\hspace{1em}(3.2.G)}$$

$$\frac{\partial \mathrm{t}\mathrm{r}({\mathbf{X}}^{\mathrm{T}}\mathbf{A}\mathbf{X})}{\partial \mathbf{X}}={\mathbf{X}}^{\mathrm{T}}(\mathbf{A}+{\mathbf{A}}^{\mathrm{T}})\text{\hspace{1em}(3.2.H)}$$

$$\frac{\partial \mathrm{t}\mathrm{r}({\mathbf{X}}^{-1}\mathbf{A})}{\partial \mathbf{X}}=-{\mathbf{X}}^{-1}{\mathbf{A}\mathbf{X}}^{-1}\text{\hspace{1em}(3.2.I)}$$

$$\frac{\partial \mathrm{t}\mathrm{r}(\mathbf{A}\mathbf{X}\mathbf{B})}{\partial \mathbf{X}}=\frac{\partial \mathrm{t}\mathrm{r}(\mathbf{B}\mathbf{A}\mathbf{X})}{\partial \mathbf{X}}=\mathbf{B}\mathbf{A}\text{\hspace{1em}(3.2.J)}$$

$$\frac{\partial \mathrm{t}\mathrm{r}({\mathbf{A}\mathbf{X}\mathbf{B}\mathbf{X}}^{\mathrm{T}}\mathbf{C})}{\partial \mathbf{X}}={\mathbf{B}\mathbf{X}}^{\mathrm{T}}\mathbf{C}\mathbf{A}+{\mathbf{B}}^{\mathrm{T}}{\mathbf{X}}^{\mathrm{T}}{\mathbf{A}}^{\mathrm{T}}{\mathbf{C}}^{\mathrm{T}}\text{\hspace{1em}(3.2.K)}$$

Reference resources

[1] https://zhuanlan.zhihu.com/p/263777564
[2] https://www.zhihu.com/question/352174717
[3] https://cloud.tencent.com/developer/article/1551901
[4] https://en.wikipedia.org/wiki/Matrix_calculus
[5] https://www.comp.nus.edu.sg/~cs5240/lecture/matrix-diff.pdf