Camera calibration (1): basic principles of monocular camera calibration and Zhang Zhengyou calibration

Why do I need to calibrate the camera

The mathematical meaning of camera ：

• The real world is three-dimensional , Taking photos is two-dimensional
• The camera （ As a generalized function ）: Input 3D scene , The output is a two-dimensional picture （ Gray value ）
• The color chart is RGB Three channels , Each channel can be considered as a gray image
• function （ The mapping relationship ） It's irreversible , That is to say, we cannot recover the three-dimensional world from two-dimensional photos ( Two dimensional photos have no depth information )
  

The significance of camera calibration

• Camera calibration ： Use a pattern Calibration board to solve the process of camera parameters
• A simplified mathematical surface model is used to represent the complex three-dimensional to two-dimensional imaging process
• Camera parameters include ： Inside the camera （ The focal length ）、 Camera external parameters （ rotate 、 Translation matrix ）, Lens distortion parameters
• purpose ： Distortion correction , Binocular vision , Structured light , Three dimensional reconstruction ,SLAM, Camera calibration is required , Only after obtaining the parameters of the camera can it be applied
  

Coordinate system transformation

Principle of pinhole imaging

Pinhole imaging instructions

• Simple without lens
• There is a small light source ( candle )
• Real world 3D object , Send light through the aperture （ Pinhole ）
• The other side of the camera , Like plane position , Get a real image of handstand
  

Introduction to coordinate system

Must know terminology ：

• World coordinate system (World Coords): The position of the point in the real world , Describe the location of the camera , The unit is m
• Camera coordinate system (Camera Coords): With the camera sensor Center as origin , Resume camera coordinate system , Company m
• Image physical coordinate system ： The two-dimensional coordinate system obtained after small hole imaging , The unit is mm, The coordinates of new year's day are the points in the graph $C$
• Pixel coordinate system (Pixel Coords): The imaging point is in the camera sensor The number of rows and columns of the upper pixel , Without any physical units
• Principal point ： Intersection of optical axis and image plane , The points in the picture p
  

In a binocular or multiocular system , The world coordinate system does not coincide with the camera coordinate system , You need to rotate the world coordinate system through the matrix R Peaceshift matrix T, To the camera coordinate system .

In the above two-dimensional plane ,$O_i$ Is the origin of the image coordinate system ,$O_d$ Is the pixel coordinate system , The pixel coordinate system is slightly offset from the origin of the image coordinate system .

(1) World coordinate system to camera coordinate system

spot p Representation in different coordinate systems

• World coordinate system (World Coords): $P(x_w,y_w,z_w)$
• Camera coordinate system (World Coords): $P(x_c,y_c,z_c)$

The transformation matrix between the world coordinate system and the camera coordinate system :

• $R$： The rotation matrix of the camera coordinate system relative to the world coordinate system
• $T$: The translation matrix of the camera coordinate system relative to the world coordinate system

Mathematical expression of transformation relation ：
$$\left[\begin{array}{c}{x}_c\\ y_c\\ z_c\\ 1\end{array}\right]=\left[\begin{array}{cc}{R}_{3\times 3}& T_{3\times 1}\\ O& 1\end{array}\right]\cdot \left[\begin{array}{c}{x}_w\\ y_w\\ z_w\\ 1\end{array}\right]$$

World coordinate system By rotating the matrix R And offset matrix T, Convert to Camera coordinate system , If the world coordinate system coincides with the camera coordinate system , be R It's an identity matrix ,T It's a zero matrix , In this way, the real world point , Convert to a point in the camera coordinate system

(2) Camera coordinate system to image coordinate system

• Suppose the point on the camera $p(x_c,y_c,z_c)$ The imaging point in the image coordinate system is $p^{{}^{\prime }}(x,y)$
• Based on the principle of small hole imaging
• A point in space is imaged in a plane , And $XcY$ Plane ( The lens ) parallel , From the origin $f$ The plane of the
• Take a section $ZcY$, You can get the right figure , The black dot in the right figure $(z_c,y_c)$, According to the similar triangle relationship, we can calculate :
  $$\frac{y}{y_c}=\frac{f}{z_c}$$
• Take a section $XcY$, According to the similar triangle relationship, we can calculate :
  $$\frac{x}{x_c}=\frac{y}{y_c}$$
• Combine two triangular transformation relations , Yes :
  $$\frac{x}{x_c}=\frac{y}{y_c}=\frac{f}{z_c}$$

After simplification, we can get :
$$x=\frac{f}{z_c}\cdot x_c$$
$$y=\frac{f}{z_c}\cdot y_c$$

• In matrix form ：
  $$z_c\cdot \left[\begin{array}{c}x\\ y\\ 1\end{array}\right]=\left[\begin{array}{cccc}f& 0& 0& 0\\ 0& f& 0& 0\\ 0& 0& 1& 0\end{array}\right]\cdot \left[\begin{array}{c}{x}_c\\ y_c\\ z_c\\ 1\end{array}\right]$$

(3) Image coordinate system to pixel coordinate system conversion

Conversion from image coordinate system to pixel coordinate system

Above picture , Image midpoint $O_b$ Represents the origin of the image coordinate system , top left corner $O_{uv}$ Represents the origin of the pixel coordinate system
Transformation of coordinate system ：

• Point of image coordinate system $p^{{}^{\prime }}(x,y)$ To the pixel coordinate system $(u,v)$ Transformation
• The origin of the image coordinate system is sensor In the middle of , The unit is mm
• The origin of the pixel coordinate system is sensor Top left corner of , Unit is Pixel, That is, the number of rows and columns of pixels
• The transformation relationship between them :
  $$u=\frac{x}{dx}+u_0,v=\frac{y}{dy}+v_0$$
• In matrix form ：
  $$\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=\left[\begin{array}{ccc}\frac{1}{dx}& 0& u_0\\ 0& \frac{1}{dy}& v_0\\ 0& 0& 1\end{array}\right]\cdot \left[\begin{array}{c}x\\ y\\ 1\end{array}\right]$$
  • $d_x$,$d_y$: yes sensor Gu you parameter , Represents the number of millimeters per pixel
  • $u_0$,$v_0$: Represents the origin of the image coordinate system （ Light heart ） The offset from the origin of the pixel coordinate system
    Sum up ： Conversion formula from camera coordinate system to pixels ：
    $$\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=\left[\begin{array}{ccc}\frac{1}{dx}& 0& u_0\\ 0& \frac{1}{dy}& v_0\\ 0& 0& 1\end{array}\right]\cdot \frac{1}{z_c}\cdot \left[\begin{array}{cccc}f& 0& 0& 0\\ 0& f& 0& 0\\ 0& 0& 1& 0\end{array}\right]\cdot \left[\begin{array}{c}{x}_c\\ y_c\\ z_c\\ 1\end{array}\right]$$
    You can get ：
    $$u=f_x\ast \frac{x_c}{z_c}+u_0$$
    $$v=f_y\ast \frac{y_c}{z_c}+v_0$$
• In the above formula :$f_x=\frac{f}{dx}$,$f_y=\frac{f}{dy}$, Focal length divided by the size of a single pixel
• During camera calibration ,$f,dx,dy$ Cannot be calibrated ,$f_x,f_y$ It can be obtained by calibration

(4) Complete coordinate system conversion

• Conversion from world coordinate system to pixel coordinate system
  $$\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=\left[\begin{array}{ccc}\frac{1}{dx}& 0& u_0\\ 0& \frac{1}{dy}& v_0\\ 0& 0& 1\end{array}\right]\cdot \left[\begin{array}{c}x\\ y\\ 1\end{array}\right]$$
  • $d_x$,$d_y$: yes sensor Gu you parameter , Represents the number of millimeters per pixel
  • $u_0$,$v_0$: Represents the origin of the image coordinate system （ Light heart ） The offset from the origin of the pixel coordinate system
    Sum up ： Conversion formula from camera coordinate system to pixels ：
    $$z_c\cdot \left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=\left[\begin{array}{ccc}\frac{1}{dx}& 0& u_0\\ 0& \frac{1}{dy}& v_0\\ 0& 0& 1\end{array}\right]\cdot \left[\begin{array}{cccc}f& 0& 0& 0\\ 0& f& 0& 0\\ 0& 0& 1& 0\end{array}\right]\cdot \left[\begin{array}{cc}{R}_{3\times 3}& T_{3\times 1}\\ O& 1\end{array}\right]\cdot \left[\begin{array}{c}{x}_w\\ y_w\\ z_w\\ 1\end{array}\right]=M_1{M}_2\left[\begin{array}{c}{x}_w\\ y_w\\ z_w\\ 1\end{array}\right]$$
• Inside the camera ： The focal length of the camera , Relative offset of pixel coordinates
  $$M_1=\left[\begin{array}{ccc}{f}_x& 0& u_0\\ 0& f_y& v_0\\ 0& 0& 1\end{array}\right]$$
• Camera external parameters ： The conversion relationship between the world coordinate system and the camera coordinate system , The pose matrix of the camera in the world coordinate system
  $$M_2=\left[\begin{array}{cc}{R}_{3\times 3}& T_{3\times 1}\end{array}\right]=\left[\begin{array}{cccc}{r}_{11}& r_{12}& r_{13}& t_1\\ r_{21}& r_{22}& r_{23}& t_2\\ r_{31}& r_{32}& r_{33}& t_3\end{array}\right]$$

Lens distortion

Lens distortion
Ultra wide angle shooting distortion will be more obvious , The more to the edge, the more obvious the distortion

• The error between the actual imaging and the ideal imaging after passing through the lens is the lens distortion
• It is mainly divided into meridional distortion and tangential distortion
  Radial distortion
• The additive lens shape results in , Along the radial distribution of the lens
• It is divided into barrel distortion and pillow distortion
• The place away from the center of the lens is more curved than the place near the center of the lens
• The distortion at the optical center is 0, The farther away from the optical center, the greater the distortion
• Cheap cameras , Abnormal changes are serious
• Mathematical polynomial description of radial distortion
  
• (x,y) It is a pixel without distortion ,$(x_{distorted},y_{distorted})$ Position after distortion
• $k_1,k_2,k_3$: Radial distortion coefficient , The internal reference of the camera , Generally, the first two items are used , Fisheye camera will use the third item

Tangential distortion

• The camera sensor Not parallel to the lens , If the camera is better, there is generally no tangential distortion . Therefore, the influence of radial distortion is generally studied .
• Mathematical representation of distortion ：
  
• The two distortions merge ：