[linear algebra] understand positive definite matrix and semi positive definite matrix

1 Definition

   In machine learning and spectral graph theory , The concept of positive definite matrix and positive semidefinite matrix is always used , In order to understand their concept is very necessary .
Definition ： Positive definite matrix （positive definite, PD）
   Given a size of $n\times n$ Real symmetric matrix of $A$, If for any length it is $n$ Nonzero vector of $X$, Yes $X^{T}AX>0$ Hang up , Then the matrix $A$ It's a positive definite matrix .

Definition ： Positive semidefinite matrices （positive semi-definite, PSD）
   Given a size of $n\times n$ Real symmetric matrix of $A$, If for any length it is $n$ Nonzero vector of $X$, Yes $X^{T}AX\ge 0$ Hang up , Then the matrix $A$ It's a positive definite matrix .
   Look at an example （ Source references 【3】）：

（1） Unit matrix $I\in {\mathbb{R}}^{2\times 2}$ Is it a positive definite matrix ？
   A vector $\mathbf{x}=\left[\begin{array}{l}{x}_1\\ x_2\end{array}\right]\in {\mathbb{R}}^2$ For the wrong $0$ vector , be

  ${\mathbf{x}}^{T}I\mathbf{x}={\mathbf{x}}^T\mathbf{x}=x_1^2+x_2^2$

   because $\mathbf{x}\ne 0$, so ${\mathbf{x}}^{T}I\mathbf{x}>0$ Hang up , So the identity matrix is a positive definite matrix .

   From the above example, we can see that positive definite matrix and semi positive definite matrix are similar to quadratic function . With a quadratic function $y=a{x}^2$ For example , The curve of this function will pass through the coordinate origin , When parameters $a>0$ when , Curved “ Opening ” Up , Parameters $a<0$ when , Curved “ Opening ” Down .
You can actually sum a quadratic function with $y=a{x}^2$ and $y=x^{T}Ax$ Compare .

• stay $y=a{x}^2$ in , if $a>0$, Then for any $x\ne 0$, Then there are $y>0$ Hang up .
  Corresponding to $y=x^{T}Ax$, if $A$ Is a positive definite matrix , Then for any $x\ne 0$, Then there are $y>0$ Hang up .
• stay $y=a{x}^2$ in , if $a\ge 0$, Then for any $x\ne 0$, Then there are $y\ge 0$ Hang up .
  Corresponding to $y=x^{T}Ax$, if $A$ Is a positive semidefinite matrix , Then for any $x\ne 0$, Then there are $y\ge 0$ Hang up .

2 Understand from a geometric point of view

   If any positive definite matrix is given $A\in R^{n\times n}$ And a non $0$ vector $x\in R^n$, Then the vector obtained by multiplying the two $y=Ax\in R^n$ And vector $x$ The included angle of is always less than $90^{.}$ Equivalent to $x^{T}Ax>0$.
   From the essence of matrix, matrix multiplication is actually a vector $x$ Installation matrix $A$ Transform in the specified way （ Matrix understanding series （ One ）（ Two ）（ 3、 ... and ））. So for a positive definite matrix $x^{T}Ax=x^{T}M>0$, remember $M=Ax$. Do you remember $Cos$ The formula
$$\cos *\mathbf{x},\mathbf{y}*=\frac{{\mathbf{x}}^T\mathbf{y}}{\Vert \mathbf{x}\Vert \cdot \Vert \mathbf{y}\Vert }$$
   Here we can understand that the function of a positive definite matrix is ： A vector $x$ Through a positive definite matrix $A$ After transformation, the original vector $x$ The included angle of is less than $90^{.}$.

Let's take another example ： Given orientation $x=\left[\begin{array}{l}2\\ 1\end{array}\right]$, For the identity matrix $I=\left[\begin{array}{ll}1& 0\\ 0& 1\end{array}\right]$, be
$\mathbf{y}=I\mathbf{x}=\mathbf{x}=\left[\begin{array}{l}2\\ 1\end{array}\right]$ Then the vector $yandx$ An Angle of 0.

$\begin{array}{l}\cos *\mathbf{x},\mathbf{y}*=\frac{{\mathbf{x}}^T\mathbf{y}}{\Vert \mathbf{x}\Vert \cdot \Vert \mathbf{y}\Vert }\\ =\frac{2\times 2+1\times 1}{\sqrt{2^2+1^2}\cdot \sqrt{2^2+1^2}}\\ =1\end{array}$
Combined with the above example and the motion of the matrix, we can understand the positive definite matrix ：

• For a vector $x$, We hope $x$ There's a matrix passing through $A$ The new vector after the change of $M$ The angle between it and itself is less than $90$ degree .
• Less than $90$ The meaning behind degree is the transformed vector $M$ It's along the original vector $x$ In the positive direction of （ namely $M$ When you project back to the original vector, the direction remains unchanged ）

   So how to understand the requirement that the eigenvalue of a positive definite matrix should be greater than 0？
   First, a matrix $A$ Eigenvector of $x$ It means that a vector will transform along the direction of the eigenvector （ The zoom ）, The scaling is determined by the eigenvalue $\lambda$ decision .（ Understanding of eigenvalues and eigenvectors ）. for instance 【 reference 【1】】：
$A_1=[[0.5,0{]}^T,[0,2{]}^T]$ It's very easy to calculate A The eigenvalues of are respectively $0.5$ and $2$, And their corresponding eigenvectors are $[1,0{]}^T$ and $[0,1{]}^T$ . So if a vector $b$ Multiply left by a matrix $A$, The essence of this is to put vectors $b$ Along $[1,0{]}^T$ and $[0,1{]}^T$ The directions are magnified separately $0.5$ and $2$ times . We assume that $b=[2,2{]}^T$, that $Ab$ The resulting vector is $[1,4{]}^T$, Combined with the figure below, it's more intuitive ：

The picture comes from the references 【1】
Let's look at the picture , If one of the eigenvalues is less than $0$, such as ${\lambda }_1<0$ So the final vector $Ab$ The vector projected to the direction is related to $b$ reverse . Sum up , To make the transformed vector $M$ And the original vector $x$ The included angle is less than $90$ degree , That is, when mapping back to the original vector, keep the direction unchanged , Then the eigenvalue needs to be greater than $0$, So that's why the eigenvalues of positive definite matrices are greater than $0$.
$$\begin{array}{c}Ax=\lambda x\\ \to x^{T}Ax=\lambda x^{T}x=\lambda \Vert x{\Vert }^2>0\end{array}$$
so λ Must be greater than 0, That is, the characteristic value must be greater than 0.
The relationship between positive definite matrix and optimization is used in the supplement .

3 reference

[1] How to understand positive definite matrix and semi positive definite matrix
[2]MIT Notes on Linear Algebra 3.3( Positive definite matrix , minimum value )
[3] Talking about 「 Positive definite matrix 」 and 「 Positive semidefinite matrices 」
[4]MIT Notes on Linear Algebra 3.1( Symmetric matrix , Positive definite matrix