Derivative Operations on Vectors and Derivative Operations on Vector Cross and Dot Products

目的：I've been writing optimized code recently,The variables in the function need to be differentiated,and find their Jacobian matrices.Therefore, the derivation of vectors and matrices is used.

A vector can be represented as follows：$Y=[y_1,y_2,...,y_m{]}^T$
Basic knowledge of vector derivatives.It is divided into the following categories：
1)向量$Y=[y_1,y_2,...,y_m{]}^T$对$x$标量求导:
$$\frac{\partial Y}{\partial x}=\left[\begin{array}{c}\frac{\partial y_1}{\partial x}\\ \frac{\partial y_2}{\partial x}\\ ⋮\\ \frac{\partial y_m}{\partial x}\end{array}\right]$$
如果$Y=[y_1,y_2,...,y_m]$是行向量,derivation
$$\frac{\partial Y}{\partial x}=\left[\begin{array}{c}\frac{\partial y_1}{\partial x}\frac{\partial y_2}{\partial x}\dots \frac{\partial y_m}{\partial x}\end{array}\right]$$

2)标量$y$对向量$X=[x_1,x_2,...,x_m{]}^T$求导
$$\frac{\partial y}{\partial X}=\left[\begin{array}{c}\frac{\partial y}{\partial x_1}\\ \frac{\partial y}{\partial x_2}\\ ⋮\\ \frac{\partial y}{\partial x_m}\end{array}\right]$$
如果$X=[x_1,x_2,...,x_m]$为行向量：
$$\frac{\partial y}{\partial X}=\left[\begin{array}{c}\frac{\partial y}{\partial x_1}\frac{\partial y}{\partial x_2}\dots \frac{\partial y}{\partial x_m}\end{array}\right]$$

3)向量$Y=[y_1,y_2,...,y_m{]}^T$对向量$X=[x_1,x_2,...,x_n]$求导
$$\frac{\partial Y}{\partial X}=\left[\begin{array}{c}\frac{\partial y_1}{\partial x_1}\frac{\partial y_1}{\partial x_2}\dots \frac{\partial y_1}{\partial x_n}\\ \frac{\partial y_2}{\partial x_1}\frac{\partial y_2}{\partial x_2}\dots \frac{\partial y_2}{\partial x_n}\\ ⋮\\ \frac{\partial y_m}{\partial x_1}\frac{\partial y_m}{\partial x_2}\dots \frac{\partial y_m}{\partial x_n}\end{array}\right]$$
Vector-to-vector derivation is also known as a Jacobian matrix,It is very common in optimization.

如果是矩阵的话,
如$Y$是矩阵的时候,它的表达：
$$Y=\left[\begin{array}{c}{y}_{11}{y}_{12}\dots y_{1n}\\ y_{21}{y}_{22}\dots y_{2n}\\ ⋮\\ y_{m1}{y}_{m2}\dots y_{mn}\end{array}\right]$$
如$X$是矩阵的时候,它的表达：
$$X=\left[\begin{array}{c}{x}_{11}{x}_{12}\dots x_{1n}\\ x_{21}{x}_{22}\dots x_{2n}\\ ⋮\\ x_{m1}{x}_{m2}\dots x_{mn}\end{array}\right]$$

There are two kinds of derivatives of matrices,如下
1)矩阵$Y$对标量$x$求导:
$$\frac{\partial Y}{\partial x}=\left[\begin{array}{c}\frac{\partial y_{11}}{\partial x}\frac{\partial y_{12}}{\partial x}\dots \frac{\partial y_{1n}}{\partial x}\\ \frac{\partial y_{21}}{\partial x}\frac{\partial y_{22}}{\partial x}\dots \frac{\partial y_{2n}}{\partial x}\\ ⋮\\ \frac{\partial y_{m1}}{\partial x}\frac{\partial y_{m2}}{\partial x}\dots \frac{\partial y_{mn}}{\partial x}\end{array}\right]$$
2)标量$y$对矩阵$X$求导:
$$\frac{\partial y}{\partial X}=\left[\begin{array}{c}\frac{\partial y}{\partial x_{11}}\frac{\partial y}{\partial x_{12}}\dots \frac{\partial y}{\partial x_{1n}}\\ \frac{\partial y}{\partial x_{21}}\frac{\partial y}{\partial x_{22}}\dots \frac{\partial y}{\partial x_{2n}}\\ ⋮\\ \frac{\partial y}{\partial x_{m1}}\frac{\partial y}{\partial x_{m2}}\dots \frac{\partial y}{\partial x_{mn}}\end{array}\right]$$
This is the derivative definition of the basic vector.Based on these definitions and some basic algorithms,Get some combined formulas.Very useful in the programming of geometric algorithms.

Derivation of vectors in formulas,In general formulas there will be multiple vectors and vector dependencies,因此,When taking the derivative, we hope that it can satisfy the chain rule of scalar derivative.
Suppose the vectors are interdependent as ：$U->V->W$
Then the partial derivative is ：
$$\frac{\partial W}{\partial U}=\frac{\partial W}{\partial V}\frac{\partial V}{\partial U}$$

证明：It is only necessary to unpack and derive the elements one by one：
$$\frac{\partial w_i}{\partial u_j}=\sum _k\frac{\partial w_i}{\partial v_k}\frac{\partial v_k}{\partial u_j}=\frac{\partial w_i}{\partial V}\frac{\partial V}{\partial u_j}$$
由此可见$\frac{\partial w_i}{\partial u_j}$is equal to the matrix$\frac{\partial W}{\partial V}$第$i$行和矩阵$\frac{\partial V}{\partial U}$的第$j$列的内积,This is the multiplication definition of a matrix.
It can easily be generalized to scenarios with multiple layers of intermediate variables.

The situation encountered in variables is often formulated as $F$为一个实数,When the intermediate variables are all vectors,它的依赖为：
$X->V->U->f$
According to the transitivity of the Jacobian matrix, it can be obtained as follows：
$$\frac{\partial F}{\partial X}=\frac{\partial F}{\partial U}\frac{\partial U}{\partial V}\frac{\partial V}{\partial X}$$
因为$f$为标量,So it is written as follows：
$$\frac{\partial f}{\partial X^T}=\frac{\partial f}{\partial U^T}\frac{\partial U}{\partial V}\frac{\partial V}{\partial X}$$
为了便于计算,The above needs to be converted to row vectors$U^T$,$X^T$计算.这个非常重要.

The following introduces the commonly used formulas encountered when calculating the reciprocal of a vector,They are of the following two categories

1）两向量$U$,$V$（列向量）The result of the dot product is pair$W$求导:
$$\frac{\partial (U^{T}V)}{\partial W}=(\frac{\partial U}{\partial W}{)}^{T}V+(\frac{\partial V}{\partial W}{)}^{T}U(4)$$
The derivative formula of the dot product proves that it is added later.
证明：假设$U=\left[\begin{array}{c}{u}_0\\ u_1\\ u_3\end{array}\right]$ 和$V=\left[\begin{array}{c}{v}_0\\ v_1\\ v_3\end{array}\right]$ ,They are three-dimensional vectors.Get the dot multiplication as $f=U^{T}V$,It is a scalar for ：$f=u_0{v}_0+u_1{v}_1+u_2{v}_2$,Then ask it to be right$W$的导数

$$\begin{gathered} \frac{\partial f}{\partial W}=\frac{\partial (u_0{v}_0+u_1{v}_1+u_2{v}_2)}{\partial W} \\ =\frac{\partial u_0}{\partial W}{v}_0+\frac{\partial v_0}{\partial W}{u}_0+\frac{\partial u_1}{\partial W}{v}_1+\frac{\partial v_1}{\partial W}{u}_1+\frac{\partial u_2}{\partial W}{v}_2+\frac{\partial v_2}{\partial W}{u}_2 \\ =(\frac{\partial u_0}{\partial W}{v}_0+\frac{\partial u_1}{\partial W}{v}_1+\frac{\partial u_2}{\partial W}{v}_2)+(\frac{\partial v_0}{\partial W}{u}_0+\frac{\partial v_1}{\partial W}{u}_1+\frac{\partial v_2}{\partial W}{u}_2) \\ =(\frac{\partial U}{\partial W}{)}^{T}V+(\frac{\partial V}{\partial W}{)}^{T}U \end{gathered}$$

It can be generalized to other dimensions.证明完毕.

如果$W$is that the scalar is actually directly substituted$(4)$即可.但是如果$W$为向量,在计算中,一般都是把$W$变成行向量.因为定义jacobi矩阵是,Derivation of a column vector with respect to a row vector.因此$W$可以表示为$W^T$(行向量),所以$(4)$,写成：
$$\frac{\partial (U^{T}V)}{\partial W^T}=(\frac{\partial U}{\partial W^T}{)}^{T}V+(\frac{\partial V}{\partial W^T}{)}^{T}U$$

2）两个向量$U$,$V$ (列向量)The result of the cross product is pair$W$求导:
$$\frac{\partial (U\times V)}{\partial W}=-Skew(V)(\frac{\partial U}{\partial W})+Skew(U)(\frac{\partial V}{\partial W})(5)$$
其中
$$Skew(U)=\left[\begin{array}{c}0-U_3{U}_2\\ U_{3}0-U_1\\ -U_2{U}_{1}0\end{array}\right]$$
其中$Skew(V)$is the matrix that converts the cross product to the dot product.It's very easy to prove,Because it is just a matrix expansion.
For cross-multiplication of multiple vectors,The formula needs to be transformed.The cross product satisfies the distribution ratio.
$$\frac{\partial (U\times V)}{\partial W}=(\frac{\partial U}{\partial W})\times V+U\times (\frac{\partial V}{\partial W})(6)$$
The proof will be added later.(5)和(6)The formulas for both are figured out.只是表达形式不同.Their transformations will be added later.

证明：假设$U=\left[\begin{array}{c}{u}_0\\ u_1\\ u_3\end{array}\right]$ 和$V=\left[\begin{array}{c}{v}_0\\ v_1\\ v_3\end{array}\right]$ ,They are three-dimensional vectors.

$$\begin{gathered} U\times V=\left[\begin{array}{c}ijk\\ u_0{u}_1{u}_2\\ v_0{v}_1{v}_2\end{array}\right] \\ =(u_1{v}_2-u_1{v}_2)i+(u_2{v}_0-u_0{v}_2)j+(u_0{v}_1-u_1{v}_0)k \end{gathered}$$

它是一个向量,因此展开后,It is expressed as follows：

$$U\times V=\left[\begin{array}{c}(u_1{v}_2-u_2{v}_1)\\ (u_2{v}_0-u_0{v}_2)\\ (u_0{v}_1-u_1{v}_0)\end{array}\right]$$

After expansion, we get the following：

$$\begin{gathered} \frac{\partial (U\times V)}{\partial W}=\left[\begin{array}{c}\frac{\partial (u_1{v}_2-u_2{v}_1)}{\partial W}\\ \frac{\partial (u_2{v}_0-u_0{v}_2)}{\partial W}\\ \frac{\partial (u_0{v}_1-u_1{v}_0)}{\partial W}\end{array}\right]=\frac{\partial (u_1{v}_2-u_2{v}_1)}{\partial W}I+\frac{\partial (u_2{v}_0-u_0{v}_2)}{\partial W}J+\frac{\partial (u_0{v}_1-u_1{v}_0)}{\partial W}K \\ =(\frac{\partial u_1}{\partial W}\ast v_2+\frac{\partial v_2}{\partial W}\ast u_1-\frac{\partial u_2}{\partial W}\ast v_1-\frac{\partial v_1}{\partial W}\ast u_2)I+(\frac{\partial u_2}{\partial W}\ast v_0+\frac{\partial v_0}{\partial W}\ast u_2-\frac{\partial u_0}{\partial W}\ast v_2-\frac{\partial v_2}{\partial W}\ast u_0)J+(\frac{\partial u_0}{\partial W}\ast v_1+\frac{\partial v_1}{\partial W}\ast u_0-\frac{\partial u_1}{\partial W}\ast v_0-\frac{\partial v_0}{\partial W}\ast u_1)K \\ =[(\frac{\partial u_1}{\partial W}\ast v_2-\frac{\partial u_2}{\partial W}\ast v_1)I+(\frac{\partial u_2}{\partial W}\ast v_0-\frac{\partial u_0}{\partial W}\ast v_2)J+(\frac{\partial u_0}{\partial W}\ast v_1-\frac{\partial u_1}{\partial W}\ast v_0)K]+[(\frac{\partial v_2}{\partial W}\ast u_1-\frac{\partial v_1}{\partial W}\ast u_2)I+(\frac{\partial v_0}{\partial W}\ast u_2-\frac{\partial v_2}{\partial W}\ast u_0)J+(\frac{\partial v_1}{\partial W}\ast u_0-\frac{\partial v_0}{\partial W}\ast u_1)K] \\ =(\frac{\partial U}{\partial W})\times V-(\frac{\partial V}{\partial W})\times U=-V\times (\frac{\partial U}{\partial W})+U\times (\frac{\partial V}{\partial W})=-Skew(V)(\frac{\partial U}{\partial W})+Skew(U)(\frac{\partial V}{\partial W}) \end{gathered}$$

the assumptions therein$a,b$为向量,Easy to get as follows
$$a\times b=-b\times a$$

It can be extended from three-dimensional to multi-dimensional vectors.证明完毕