Introduction Guide to stereo vision (3): Zhang calibration method of camera calibration [ultra detailed and worthy of collection]

Dear students , Our world is 3D The world , Our eyes can observe three-dimensional information , Help us perceive distance , Navigation obstacle avoidance , So as to soar between heaven and earth . Today's world is an intelligent world , Our scientists explore a variety of machine intelligence technologies , Let the machine have the three-dimensional perception of human beings , And hope to surpass human beings in speed and accuracy , For example, positioning navigation in autopilot navigation , Automatic obstacle avoidance of UAV , Three dimensional scanning in the measuring instrument, etc , High intelligent machine intelligence technology is 3D Visual realization .

Stereo vision is an important direction in the field of 3D reconstruction , It simulates the structure of the human eye and simulates the binocular with two cameras , Project in Perspective 、 Based on triangulation , Through the logical complex homonymous point search algorithm , Restore the 3D information in the scene . It is widely used , Autopilot 、 Navigation obstacle avoidance 、 Cultural relic reconstruction 、 Face recognition and many other high-tech applications have its key figure .

This course will help you to understand the theoretical and practical knowledge of stereo vision from simple to profound . We will talk about the coordinate system and camera calibration , From passive stereoscopic to active stereoscopic , Even from deep recovery to grid construction and processing , Interested students , Come and explore the charm of stereovision with me ！

This course is an electronic resource , So there will not be too many restrictions in the writing , But it will be logically clear 、 Easy to understand as the goal , Level co., LTD. , If there are deficiencies , Please give me some advice ！
Personal wechat ：EthanYs6, Add me to the technical exchange group StereoV3D, Talk about technology together .
CSDN Search for ：Ethan Li Li Yingsong , see Web based courses .
Lesson code , Upload to github On , Address ：StereoV3DCode：https://github.com/ethan-li-coding/StereoV3DCode

Hello, students , In the last article Introduction to stereo vision （2）： Key matrix in , Our understanding of stereovision 3 Key matrix ： The essential matrix $E$、 Basic matrix $F$、 Homography matrix $H$ A more detailed description is given , At the same time, the essential matrix is given 、 The solution method of homography matrix and the external parameters of essential matrix decomposition $R,t$ The specific formula . What's more valuable is , At the end of the blog, we provided several homework questions and put them in Github Open source reference answer code 【 I know a lot of psychology are meditating on Li Bo 666】【 Of course, there must be some people meditating on this too easy I wonder if Li Bo can have some difficulty 】. in any case , Bloggers think this is a meaningful thing , I just hope it doesn't hurt people's children .

And the content of this article , Is the absolute core module of stereo vision ： Camera calibration . Although it is relatively mature because of its technology , Nowadays, few people study , It is also easy to be ignored , Often use an open source algorithm library, such as Opencv perhaps Matlab Calibrate the toolbox directly , But in reality, stereovision Engineering 、 When commercializing , Open source tools because of their low accuracy 、 Low flexibility and not recommended for direct use , Enterprises often develop camera calibration algorithms by themselves . Camera calibration is the core module of stereo vision , It is quite necessary to master its theory , It is very helpful for us to deeply understand stereo vision technology , It has the effect of opening mountains and roads .

Camera calibration , The purpose is to determine the camera Internal parameter matrix $K$、 External parameter matrix $R,t$ and Distortion coefficient $d(k_1,k_2,k_3,p_1,p_2)$. Let's not consider the distortion coefficient for the moment （ We will discuss it later ）, As we know from the previous chapter , Image coordinates $p$ And world coordinates $P_w$ Between , Establish projection relationship through internal and external parameters , The formula is as follows ：
$$\lambda p=K\left[\begin{array}{cc}R& t\end{array}\right]\left[\begin{array}{c}{P}_w\\ 1\end{array}\right]\text{\hspace{1em}(1)}$$
The so-called calibration , That is, the process of fitting a parametric model from a large number of observations , And the parameter model fitted here is known , Therefore, we should try our best to explore a scheme that can easily obtain a large number of observations , If some other geometric constraints are satisfied between the observed values, it is more helpful to solve the specific single parameter values .

What we are talking about today Zhang Formula calibration method ¹ That is, it provides a convenient scheme to obtain a large number of observations , At the same time, the observed values also meet a class of obvious geometric constraints （ Plane constraint ）, Internal and external parameters can be directly solved . Its operation mode is very simple , Just shoot a plane with a calibration plate pattern , Camera calibration can be completed , The difficulty of calibration is greatly reduced , If we don't pursue high precision , Print a checkerboard calibration board pattern and paste it on a nearly flat cardboard to complete the calibration , Speed up the introduction and popularization of stereo vision , Far reaching , It is an absolute classic in the field of camera calibration .

This article will take you to know more about Zhang Camera calibration method , Mastering this article is very necessary for mastering the theory of stereo vision and engineering practice .

Implementation method

Zhang The formula calibration method can be widely used , One important reason is that its implementation method is very simple , It can be completed without professional process .

First step , Design a sheet with obvious corner features , And the pattern of two-dimensional coordinates of each corner is known as the calibration pattern , There are three common patterns ：

Chessboard

ChArUco

Dot

Regular pattern design can easily calculate the two-dimensional coordinates of corners in the pattern , Take checkerboard , The number of pixels between corners is fixed , Suppose the coordinates of the upper left corner are $(0,0)$, Then the pixel coordinates of other corners can be calculated by the offset of the grid , And a known DPI Calibration board image of , After printing, the two-dimensional space coordinates of each corner are also completely known （ Convert into space size through pixels ）.

The second step , Place the calibration plate pattern on a plane in some way . For example, the simplest way is to print out the original size of the calibration chart , Then find an approximately flat plate , Paste the printed calibration pattern onto the tablet ; A more professional and high-precision way is to find professional manufacturers to make high-precision flat plates （ Such as ceramic plate ） And the calibration pattern is engraved on the flat plate by some process . The purpose of this step is to make the corners of the calibration pattern lie on a plane .

picture source ：https://calib.io/blogs/knowledge-base/calibration-patterns-explained

such , The two-dimensional coordinates described in the first step can be converted into the third dimension $Z$ The coordinates are equal to 0 Three dimensional coordinates of （ Place the origin of the world coordinate system at a corner of the calibration plate ,Z The axis is perpendicular to the calibration plate ）.

The third step , Move the camera to N（N>=3） Shoot the pattern of the calibration board with different poses .

Step four , Extract the corners of the calibration plate pattern taken in the previous step , Calculate the calibration parameters .

matlab Schematic diagram of software calibration

The above is the implementation steps of camera calibration , In conclusion , There is a set of corner points with known spatial coordinates on a plane calibration plate , The camera takes the corner pattern under multiple different poses and extracts the pixel coordinates of the corner , You can solve the internal and external parameters of the camera . This summary contains 2 An important knowledge point ：

1. The spatial coordinates of the corners in the calibration pattern are known , And they are all on the same plane ,$Z$ The coordinates are equal to 0
2. The camera needs to take corner patterns in multiple different poses and extract pixel coordinates

You can see Zhang The formula calibration method is indeed an easy method to implement , The calibration theory contained therein , It's very valuable , For beginners of stereo vision , It is necessary to master its theory , Please also be sure to read the rest of this article .

Theoretical basis

Definition

In order to keep consistent with the formula in the paper , Let's change the naming habits in the first two blogs , Suppose the pixel coordinates of a point on the image are $\text{m}=[u,v{]}^T$, The three-dimensional coordinates of a point in space are $\text{M}=[X,Y,Z{]}^T$, The homogeneous expressions of the two are $\tilde{\text{m}}=[u,v,1{]}^T,\tilde{\text{M}}=[X,Y,Z,1{]}^T$. The projection formula of the title of the article can be expressed as ：
$$s\tilde{\text{m}}=\text{A}\left[\begin{array}{cc}\text{R}& \text{t}\end{array}\right]\tilde{\text{M}}\text{\hspace{1em}(2)}$$
among ,$s$ Is the scale factor ,$\text{R},\text{t}$ External parameter matrix , Used to convert world coordinate system coordinates into camera coordinate system coordinates , The former is the rotation matrix , The latter is the translation vector .$\text{A}$ Is the camera's internal parameter matrix , Concrete ,
$$A=\left[\begin{array}{ccc}\alpha & \gamma & u_0\\ 0& \beta & v_0\\ 0& 0& 1\end{array}\right]$$
among ,$(u_0,v_0)$ Is the coordinate of the image principal point ,$\alpha ,\beta$ They are images $u$ Axis and $v$ Focal length in axial direction （ Pixel unit ） value ,$\gamma$ Is the tilt factor of the pixel axis .

Homography matrix

In the implementation method , We must be able to pay attention to a very critical message ： The calibration pattern is placed on a plane . Its purpose is to make the corners in the calibration pattern lie on a spatial plane , So when the camera takes corner imaging , There is a homography transformation relationship between the space plane and the image plane , Through a homography matrix $\text{H}$ Convert the spatial coordinates of corners into image coordinates .

In general , We place the world coordinate system on a corner of the calibration plate , And let $Z$ The axis is perpendicular to the calibration plate , At this point $Z$ The coordinates will all be equal to 0, Then the three-dimensional coordinates can be expressed as $[X,Y,0,1{]}^T$. Also define the rotation matrix $\text{R}$ Of the $i$ The column vectors are ${\text{r}}_i$, The formula $s\tilde{\text{m}}=\text{A}\left[\begin{array}{cc}\text{R}& \text{t}\end{array}\right]\tilde{\text{M}}$ Can be expressed as
$$s\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=\text{A}\left[\begin{array}{cccc}{\text{r}}_1& {\text{r}}_2& {\text{r}}_3& \text{t}\end{array}\right]\left[\begin{array}{c}X\\ Y\\ 0\\ 1\end{array}\right]=\text{A}\left[\begin{array}{ccc}{\text{r}}_1& {\text{r}}_2& \text{t}\end{array}\right]\left[\begin{array}{c}X\\ Y\\ 1\end{array}\right]\text{\hspace{1em}(3)}$$
As mentioned earlier , Homography matrix can be used between calibration plane and image plane $\text{H}$ To transform , That is to say
$$s\tilde{\text{m}}=\text{H}\tilde{\text{M}}\text{\hspace{1em}(4)}$$
Due to the space point $Z$ Coordinate for 0, So in the above formula $\tilde{\text{M}}=[X,Y,1{]}^T$

Combined （3）, Available
$$\text{H}=\lambda \text{A}\left[\begin{array}{ccc}{\text{r}}_1& {\text{r}}_2& \text{t}\end{array}\right]\text{\hspace{1em}(5)}$$
$\text{H}$ It's a $3\times 3$ Matrix ,$\lambda$ Is an arbitrary scalar （$\lambda$ Because of the scale invariance of homogeneous coordinates , You can also argue that $\lambda$ Is equal to 1）.

Constraints on internal parameters

Bearing on the , We define $\text{H}=\left[\begin{array}{ccc}{\text{h}}_1& {\text{h}}_2& {\text{h}}_3\end{array}\right]$, Then there are
$$\left[\begin{array}{ccc}{\text{h}}_1& {\text{h}}_2& {\text{h}}_3\end{array}\right]=\lambda \text{A}\left[\begin{array}{ccc}{\text{r}}_1& {\text{r}}_2& \text{t}\end{array}\right]\text{\hspace{1em}(6)}$$

Because of rotation $\text{R}$ A matrix is an orthogonal matrix ,${\text{r}}_1$ and ${\text{r}}_2$ It's a standard orthogonal basis , That is, they are not only perpendicular to each other , And the mold length is 1. The meet ${\text{r}}_1^T{\text{r}}_2=0$,${\text{r}}_1^T{\text{r}}_1={\text{r}}_2^T{\text{r}}_2=1$.

And the above formula can be deduced ${\text{r}}_1=\frac{1}{\lambda }{\text{A}}^{-1}{\text{h}}_1$,${\text{r}}_2=\frac{1}{\lambda }{\text{A}}^{-1}{\text{h}}_2$.

It's easy to get ：
$$\begin{array}{rl}{\text{h}}_1^T{\text{A}}^{-T}{\text{A}}^{-1}{\text{h}}_2& =0\\ & \\ {\text{h}}_1^T{\text{A}}^{-T}{\text{A}}^{-1}{\text{h}}_1& ={\text{h}}_2^T{\text{A}}^{-T}{\text{A}}^{-1}{\text{h}}_2\end{array}\text{\hspace{1em}(7)}$$
It can be seen from the above formula that , Homography matrix $\text{H}$ And internal parameter matrix $\text{A}$ The elements of satisfy the constraints of two linear equations . If we can calculate the homography matrix , Intuitively, it feels that the internal parameter matrix can be solved by solving the linear equation . Is it possible to be specific ？ Let's move on .

Camera parameter solution

This section will take you to the specific solution of camera parameters , Answer the questions left over from the previous section , That is, whether it can pass the formula （7） To solve camera parameters .

This section will be divided into 3 Parts of , The first part introduces the direct by formula （7） Closed form solution formula for solving camera parameters ; The second part introduces the maximum likelihood solution of camera parameters ; The third part introduces the solution of distortion parameters after considering camera distortion .

Closed solution

First define $\text{B}={\text{A}}^{-T}{\text{A}}^{-1}=\left[\begin{array}{ccc}{B}_{11}& B_{12}& B_{13}\\ B_{12}& B_{22}& B_{23}\\ B_{13}& B_{23}& B_{33}\end{array}\right]$.

Concrete , You can calculate that $\text{B}$ Each element of ：
$$\text{B}=\left[\begin{array}{ccc}\frac{1}{{\alpha }^2}& -\frac{\gamma }{{\alpha }^2\beta }& \frac{v_0\gamma -u_0\beta }{{\alpha }^2\beta }\\ -\frac{\gamma }{{\alpha }^2\beta }& \frac{{\gamma }^2}{{\alpha }^2{\beta }^2}+\frac{1}{{\beta }^2}& -\frac{\gamma (v_0\gamma -u_0\beta )}{{\alpha }^2{\beta }^2}-\frac{v_0}{{\beta }^2}\\ \frac{v_0\gamma -u_0\beta }{{\alpha }^2\beta }& -\frac{\gamma (v_0\gamma -u_0\beta )}{{\alpha }^2{\beta }^2}-\frac{v_0}{{\beta }^2}& \frac{(v_0\gamma -u_0\beta {)}^2}{{\alpha }^2{\beta }^2}+\frac{v_0^2}{{\beta }^2}+1\end{array}\right]\text{\hspace{1em}(8)}$$
（ It is suggested that students should work out the formula （8）, Although it looks complicated , But the derivation is actually very simple ）

obviously ,$\text{B}$ It's a symmetric matrix , We can use one 6 The vector of dimension ：
$$\text{b}={\left[\begin{array}{cccccc}{B}_{11}& B_{12}& B_{22}& B_{13}& B_{23}& B_{33}\end{array}\right]}^T\text{\hspace{1em}(9)}$$
Define homography matrix $\text{H}$ Of the $i$ The column vector is ${\text{h}}_i={\left[\begin{array}{ccc}{h}_{i1}& h_{i2}& h_{i3}\end{array}\right]}^T$, Then there are
$${\text{h}}_i^T\text{B}{\text{h}}_j={\text{v}}_{ij}^T\text{b}\text{\hspace{1em}(10)}$$
among ,${\text{v}}_{ij}={\left[\begin{array}{cccccc}{h}_{i1}{h}_{j1}& h_{i1}{h}_{j2}+h_{i2}{h}_{j1}& h_{i2}{h}_{j2}& h_{i3}{h}_{j1}+h_{i1}{h}_{j3}& h_{i3}{h}_{j2}+h_{i2}{h}_{j3}& h_{i3}{h}_{j3}\end{array}\right]}^T$.

successively , The formula (7) Can be converted to ：
$$\left[\begin{array}{c}{\text{v}}_{12}^T\\ ({\text{v}}_{11}-{\text{v}}_{22}{)}^T\end{array}\right]\text{b}=0\text{\hspace{1em}(11)}$$

This is a typical system of linear equations , The number of rows of the coefficient matrix is 2, solve 6 Dimension vector $\text{b}$. By formula （4）, When the camera is 1 After shooting the pattern of the calibration board in three positions , The pixel coordinates of corners are extracted , The corresponding relationship between the world coordinate system and the pixel coordinate system of all corners can be obtained , Then the homography transformation matrix under the current pose is solved by the least square method of linear equations $\text{H}$, We can get the formula （11） The concrete expression of .

But the formula （11） The coefficient matrix of is only 2 That's ok , Demand solution 6 Dimension vector $\text{b}$ It's not enough. . So we need a camera in $n$ Shoot the calibration pattern in three positions , obtain $n$ A homography matrix , And the number of lines is $2n$ The coefficient matrix of , When $n>=3$ when , It can be solved 6 Dimension vector $\text{b}$. That is, at least 3 The camera calibration can only be completed with pictures . The final total equations can be expressed as ：
$$\text{V}\text{b}=0\text{\hspace{1em}(12)}$$

$\text{V}$ It's a $2n\times 6$ The coefficient matrix of .

Solve the formula （12） Common methods , One is matrix ${\text{V}}^T\text{V}$ The eigenvector corresponding to the minimum eigenvalue of is the solution of the equation , The second is the coefficient matrix $\text{V}$ Singular value decomposition $\text{V}=USV$, Decomposition matrix $V$ The third column of is the solution of the equation .

It should be noted that , The formula （12） Is scale equivalent , Solved $\text{b}$ Multiplying by any multiple is still the correct solution , So by the $\text{b}$ Matrix of composition $\text{B}$ Not strictly satisfied $\text{B}={\text{A}}^{-T}{\text{A}}^{-1}$, But there is an arbitrary scale factor $\lambda$, Satisfy $\text{B}=\lambda {\text{A}}^{-T}{\text{A}}^{-1}$. By derivation , The formula can be used by （13） To calculate all camera internal parameters ：
$$\begin{array}{rl}{v}_0& =(B_{12}{B}_{13}-B_{11}{B}_{23})/(B_{11}{B}_{22}-B_{12}^2)\\ \lambda & =B_{33}-[B_{13}^2+v_0(B_{12}{B}_{13}-B_{11}{B}_{23})]/B_{11}\\ \alpha & =\sqrt{\lambda /B_{11}}\\ \beta & =\sqrt{\lambda B_{11}/(B_{11}{B}_{22}-B_{12}^2)}\\ \gamma & =-B_{12}{\alpha }^2\beta /\lambda \\ u_0& =\gamma v_0/\beta -B_{13}{\alpha }^2/\lambda \end{array}\text{\hspace{1em}(13)}$$
When $\text{A}$ After solving , External parameters under each pose $\text{R},\text{t}$ You can use the formula （14） Calculation ：
$$\begin{array}{rl}{\text{r}}_1& =\lambda {\text{A}}^{-1}{\text{h}}_1\\ {\text{r}}_2& =\lambda {\text{A}}^{-1}{\text{h}}_2\\ {\text{r}}_3& ={\text{r}}_1\times {\text{r}}_2\\ \text{t}& =\lambda {\text{A}}^{-1}{\text{h}}_3\end{array}\text{\hspace{1em}(14)}$$
among ,$\lambda =1/||{\text{A}}^{-1}{\text{h}}_1||=1/||{\text{A}}^{-1}{\text{h}}_2||$.

It should be noted that , Because of the noise , Calculated rotation matrix $\text{Q}=[{\text{r}}_1,{\text{r}}_2,{\text{r}}_3]$ Does not satisfy orthogonality , Further calculations and $\text{Q}$ The nearest orthogonal matrix $\text{R}$, It can be calculated by singular value decomposition . The best orthogonal matrix $\text{R}$ The matrix should be satisfied $\text{R}-\text{Q}$ Of Frobenius Norm minimum , That is to solve the following problem ：
$$\underset{\text{R}}{\min }||\text{R}-\text{Q}|{|}_F^2\text{subject to}{\text{R}}^T\text{R}=\text{I}\text{\hspace{1em}(15)}$$
because
$$\begin{array}{rl}||\text{R}-\text{Q}|{|}_F^2& =trace((\text{R}-\text{Q}{)}^T(\text{R}-\text{Q}))\\ & =3+trace({\text{Q}}^T\text{Q})-2trace({\text{R}}^T\text{Q})\end{array}$$
therefore , Turn the problem into a solution $trace({\text{R}}^T\text{Q})$ maximal $\text{R}$.
We define the matrix $\text{Q}$ The singular value of is decomposed into $\text{U}\text{S}{\text{V}}^T$, among $\text{S}=diag({\sigma }_1,{\sigma }_2,{\sigma }_3)$. If we define an orthogonal matrix $\text{Z}={\text{V}}^T{\text{R}}^T\text{U}$, Then there are
$$\begin{array}{rl}trace({\text{R}}^T\text{Q})& =trace({\text{R}}^T\text{U}\text{S}{\text{V}}^T)=trace({\text{V}}^T{\text{R}}^T\text{U}\text{S})\\ & =trace(\text{Z}\text{S})=\sum _{i=1}^3{z}_{ii}{\sigma }_i\le \sum _{i=1}^3{\sigma }_i\end{array}$$
obviously , When $\text{Z}=\text{I}$ when ,$trace({\text{R}}^T\text{Q})$ To the maximum , Available $\text{R}=\text{U}{\text{V}}^T$.

above , All internal and external parameters are solved , The whole closed solution is completed .

Maximum likelihood estimation

The solution with the minimum algebraic distance can be obtained by the closed solution , The actual geometric meaning of each parameter is not taken into account , Because of the noise , The solution is not very accurate . We can obtain a more accurate solution by maximum likelihood estimation .

If we observe the calibration plate $n$ In the shot image $m$ A point is right . Suppose that all points have independent equal scale noise , Then the maximum likelihood estimation minimizes the following expression ：
$$\sum _{i=1}^n\sum _{j=1}^m||{\text{m}}_{ij}-\hat{\text{m}}(\text{A},{\text{R}}_i,{\text{t}}_i,{\text{M}}_j)|{|}^2\text{\hspace{1em}(16)}$$
among ,$\hat{\text{m}}(\text{A},{\text{R}}_i,{\text{t}}_i,{\text{M}}_j)$ It's a space point ${\text{M}}_j$ In the image $i$ The point of projection on , By formula （2） Calculation . Rotation matrix $\text{R}$ Through three-dimensional vector $\text{r}$ To express ,$\text{r}$ It is parallel to the rotation axis and the die length is the rotation angle ,$\text{R}$ and $\text{r}$ It can be converted to each other through rodry format formula .

type （16） Is a nonlinear expression , So this is a nonlinear minimization problem , It can be used Levenberg-Marquardt Algorithm to solve . Presumably, some students know that solving this kind of problem requires a relatively accurate initialization value , This value can be obtained by using the closed solution obtained in the previous section .

In engineering practice , We usually use ceres-solver Library to handle this part .

Camera distortion

up to now , We haven't considered camera distortion , All derivations are based on the ideal case without distortion , The actual situation is that the camera will inevitably have more or less distortion , There are mainly two kinds of ： Radial distortion and tangential distortion . In general , Will consider 3 Radial distortion of the term $k_1,k_2,k_3$ and 2 Tangential distortion of the term $p_1,p_2$. stay Zhang In the formula calibration method , Only considered 2 Radial distortion of the term $k_1,k_2$, And in practice , We will consider more , The same principle , And so on .

Definition $(u,v)$ Is the ideal pixel coordinate ,$(\breve{u},\breve{v})$ Is the actually observed pixel coordinates , Consider 2 After radial distortion , Both satisfy the following formula ：
$$\begin{array}{rl}\breve{u}& =u+(u-u_0)[k_1(x^2+y^2)+k_2(x^2+y^2{)}^2]\\ \breve{v}& =v+(v-v_0)[k_1(x^2+y^2)+k_2(x^2+y^2{)}^2]\end{array}\text{\hspace{1em}(17)}$$
among ,$x=u-u_0$,$y=v-v_0$.

type （17） It's a system of linear equations , solve $k_1,k_2$ It seems to be a simple thing , But in fact, there is a problem , That is, it is impossible to obtain the ideal coordinates without distortion $(u,v)$, It needs to know the accurate internal and external parameters through the projection formula （2） To solve , Students may ask ： In the first two sections, we solved the internal and external parameters ！？ Don't forget to , The camera distortion is not considered in the front , The solution of internal and external parameters is actually inaccurate （ The observed value is regarded as the internal and external parameters of distortion free coordinates , It's biased ）. So this has become a contradiction between chicken and egg .

Then simply do not expect to calculate the exact distortion through the linear equation , The internal and external parameters obtained from the closed solution are substituted into the formula （2） Find the approximate ideal coordinates $(u,v)$, Thus, the formula （17） Establish linear equations to solve approximate $k_1,k_2$.

then , We establish a new maximum likelihood estimation expression ：
$$\sum _{i=1}^n\sum _{j=1}^m||{\text{m}}_{ij}-\breve{\text{m}}(\text{A},k_1,k_2,{\text{R}}_i,{\text{t}}_i,{\text{M}}_j)|{|}^2\text{\hspace{1em}(18)}$$
Similarly, all internal and external parameters and distortion coefficients are solved by nonlinear solution method . The initial value of the distortion coefficient is solved by the method described above .

actually , Because the distortion coefficient is very small , Therefore, the initial values of the distortion coefficients are directly set to 0 It's OK, too , There is no need to solve linear equations .

summary

Next to the whole Zhang Make a summary of the calibration process ：

1. Print the calibration pattern and paste it on a plane , Call it a calibration plate .
2. By moving the camera or the calibration plate, multiple calibration plate images are taken in different positions and attitudes （ Number of images >=3）.
3. Detect feature points on all images （ Corner or center point ）.
4. Use the closed solution to solve all internal and external parameters .
5. Approximate distortion coefficients are solved by linear equations （ Or directly assign a value of 0）.
6. Accurate internal and external parameters and distortion coefficients are calculated through nonlinear optimization .

The above is Zhang All theoretical explanations of the formula calibration method , I hope students can be enlightened after reading this article , And be able to understand , It is handy in practical engineering application .

stay Zhang In the original text of formula calibration method , There are also some high-level knowledge points worth learning that are not reflected in this article , Including key formulas （7） The geometric interpretation of , And the discussion of degradation , Interested students can continue to read the original text to understand .

In the next article, we will bring another method of camera calibration ：DLT Direct linear transformation .

Homework

Here are some exercises for you , We can deepen our understanding through practice ：

1 adopt opencv The interface provided by the open source library completes camera calibration , You can use opencv Images provided , You can also use your own image .
2 Higher order is , You can be independent opencv Library to write a camera calibration algorithm ？ Or just use opencv To detect the corner coordinates , Other steps are implemented by yourself .

Please refer to the address ：https://github.com/ethan-li-coding/StereoV3DCode [ The code is constantly updated , If you are interested, you can star and watch, This code is temporarily missing ]

reference

- [1] Zhang Z . A Flexible New Technique for Camera Calibration[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2000, 22(11):1330-1334.