Scalar / vector / matrix derivation method

This blog stems from the derivation of an error vector Euclidean distance when reading the paper , as follows ：

After reading a pile of data, I came to the following conclusion ：

How did this conclusion come about ？ This involves scalars / vector / The derivation of the matrix . Due to scalar 、 Vectors can be regarded as special matrices , Therefore, it is collectively referred to as matrix derivation .

Catalog

1、 How to derive a matrix

2、 The denominator layout of matrix derivation

3、 Common formulas for denominator layout derivation

4、 Molecular arrangement

5、 Jacobian matrix And Jacobian determinant

---

1、 How to derive a matrix

For general scalar functions f(x) For scalars x The derivative of is ：

  matrix A The matrix B The derivation of dA/dB It is essentially a matrix A Medium Every element For the matrix B Medium Every element The derivation of . According to this essence, we can get the number of derivatives in the following cases .

1、 When A And B All are 1×1 Matrix time , here A Yes B Derivation is the derivation of scalar function from scalar , The result is 1 A derivative ;

 2、 When A by m×1 Vector ,B by 1×1 Scalar time of , here A Yes B Derivation is the derivation of a vector function from a scalar , There are m A derivative ;

3、 When A by m×1 Vector ,B by p×1 When the vector of , Such as ：

  There are m×p A derivative ;

4、 When A by m×n Vector ,B by p×q When the vector of , There are m×n×p×q A derivative

In the above four cases , Except that the first result is a scalar , The other three derivatives have multiple . In practical applications , We often expect that the result of matrix derivation is also in the form of a matrix to participate in subsequent operations . By the end of 4 For example , The final m×n×p×q How should the derivatives be arranged in matrix form ？ Is arranged in m×n That's ok p×q Or m That's ok n×p×q Columns or other forms ？ This leads to the matrix （ vector ） Derivative Molecular arrangement And Denominator layout 了 .

Molecular arrangement And Denominator layout In essence, it is the layout of two artificially prescribed derivative results , There is no difference between good and bad , The specific layout can be selected according to the calculation habits . For the derivative dA/dB , Molecular arrangement Is to find the derivative result in terms of the matrix at the molecular position A The form shall prevail （ The first dimension of the derivative result is consistent with the molecular position ）, Denominator layout Is the matrix on the denominator of the derivative result B The form shall prevail . Let's say Denominator layout Take an example to explain .

2、 The denominator layout of matrix derivation

For the derivative  df/dx Come on ,f At the molecular position ,x In the denominator position , The discussion can be divided into the following situations ：

（ Generally, the vector represented by letters defaults to Column vector , Unless transposed ）

1、f  Is a scalar function ,x  Is the column vector ：

  here , Because the denominator part is a p×1 Column vector form of , So the final result is p×1 The column vector .

2、f  Is a scalar function ,x  It's a row vector ：

  here , Because the denominator part is a 1×p In the form of a row vector , So the final result is 1×p The row vector .

3、f  Is the column vector ,x  For the scalar ：

  The result is 1×p In the form of a row vector , This result is equivalent to

 4、f  Is the column vector ,x  Is the column vector ：

At this time, the form of the final result can be deduced in two steps .

First, the first step will be f As a whole , According to the position of the denominator x The layout of

  Now you can put According to the situation 3 The form in is expanded as follows ：

For the denominator layout, the final result is consistent with the form of the denominator , There is a simple image called XY Stretching , The form of the result can be judged by the following two criteria ：

• Scalar invariant , Vector stretch
• X Longitudinal stretch ,Y Transverse stretch

The stretching explanation of the above formula is shown in the figure below ：

The above is matrix derivation Denominator layout Several situations of , Other complex combinations can be derived from the above rules . The denominator layout is only an artificial layout regulation of the matrix derivation results , The derivative result is just a few definite values , In specific applications, we layout these values into one-dimensional vectors 、 Two dimensional matrix or higher dimensional form are set according to their own needs , Just the denominator / The form of molecular layout is more commonly used in general applications .

3、 Common formulas for denominator layout derivation

（1）

  among ：

  This problem can be looked at first f(x) The expansion of , Notice that the result is Scalar ：

  According to the rules of denominator layout , The derivative result at this time should be a p×1 The column vector , here

（2）

among ：

 

  You can launch at this time ：

（3） This is also the calculation mentioned at the beginning of the article （ It can be deduced by expanding first and then deriving each item ）

（4）

（5）

4、 Molecular arrangement

For molecular layout, the final result is consistent with the form of molecules , Similar to the denominator layout XY Stretching , There is also a simple and easy to remember method of molecular layout, namely YX Stretching , The form of the result can be judged by the following criteria ：

• Scalar invariant , Vector stretch
• Y Longitudinal stretch ,X Transverse stretch （ Note that the denominator layout is X Longitudinal stretch ,Y Transverse stretch ）

Generally speaking, the results of numerator layout and denominator layout are transposed to each other . There is no difference between the two layouts , Just find a way to use it that you are used to .

5、 Jacobian matrix And Jacobian determinant

Matrix by Molecular arrangement The result of derivation constitutes Jacobian matrix （ The result of denominator layout is the transposition of Jacobian determinant ）. When the Jacobian matrix is a square matrix , The determinant of the square matrix can be called Jacobian determinant . In matrix theory , The determinant of the linear transformation matrix represents the change multiple of the space area before and after the transformation , But for nonlinear transformation matrix , The establishment of its Jacobian determinant is to solve the problem of local space expansion , use Infinitesimal method （ Infinitesimal transformation can actually be regarded as linear ） Represents the multiple of space expansion near a certain point .

===================================================================

of The geometric principle of matrix linear transformation as well as The geometric principle of Jacobian determinant   You can refer to the following video ：

【 As the saying goes, matrix 】 The essential meaning of determinant is actually like this ！ The math teacher will never tell you ！_ Bili, Bili _bilibilihttps://www.bilibili.com/video/BV1Hb4y1q7Sp?from=search&seid=9625805080710138609&spm_id_from=333.337.0.0《 Under Jacobian matrix ： The so-called Jacobian determinant 》3Blue1Brown Grant Sanderson, Moved from Khan College . 【 Self made Chinese subtitles 】_ Bili, Bili _bilibilihttps://www.bilibili.com/video/BV18J41157X8/?spm_id_from=333.788.recommend_more_video.12

Matrix derivation Reference video ：

Introduction to econometrics 2—— Matrix derivation （ Denominator layout ）【 On 】_ Bili, Bili _bilibilihttps://www.bilibili.com/video/BV1fK411W7oh/?spm_id_from=333.788.recommend_more_video.0