Introduction Guide to stereo vision (5): dual camera calibration [no more collection, I charge ~]

Dear students , Our world is 3D The world , Our eyes can observe three-dimensional information , Help us perceive distance , Navigation obstacle avoidance , So as to soar between heaven and earth . Today's world is an intelligent world , Our scientists explore a variety of machine intelligence technologies , Let the machine have the three-dimensional perception of human beings , And hope to surpass human beings in speed and accuracy , For example, positioning navigation in autopilot navigation , Automatic obstacle avoidance of UAV , Three dimensional scanning in the measuring instrument, etc , High intelligent machine intelligence technology is 3D Visual realization .

Stereo vision is an important direction in the field of 3D reconstruction , It simulates the structure of the human eye and simulates the binocular with two cameras , Project in Perspective 、 Based on triangulation , Through the logical complex homonymous point search algorithm , Restore the 3D information in the scene . It is widely used , Autopilot 、 Navigation obstacle avoidance 、 Cultural relic reconstruction 、 Face recognition and many other high-tech applications have its key figure .

This course will help you to understand the theoretical and practical knowledge of stereo vision from simple to profound . We will talk about the coordinate system and camera calibration , From passive stereoscopic to active stereoscopic , Even from deep recovery to grid construction and processing , Interested students , Come and explore the charm of stereovision with me ！

This course is an electronic resource , So there will not be too many restrictions in the writing , But it will be logically clear 、 Easy to understand as the goal , Level co., LTD. , If there are deficiencies , Please give me some advice ！
Personal wechat ：EthanYs6, Add me to the technical exchange group StereoV3D, Talk about technology together .
CSDN Search for ：Ethan Li Li Yingsong , see Web based courses .
Lesson code , Upload to github On , Address ：StereoV3DCode：https://github.com/ethan-li-coding/StereoV3DCode

The first two bloggers introduced two single camera calibration methods , Some students will have doubts , This series is clearly stereo vision , Why introduce single camera calibration instead of dual camera calibration ？ The reason is simple , That is, single camera calibration is the basis of dual camera calibration , say concretely , Dual camera calibration is to complete the calibration of two single cameras respectively , Then conduct the overall calibration . And listen to me ~

Binocular stereo

Have a common sense cognition , That is, stereo vision generally refers to a stereo system composed of two cameras , Just like the human eye . When there is an overlapping area between the views taken by two cameras , They form a basic binocular stereo .

Double view through epipolar correction and stereo matching , You can get three-dimensional points through triangular relations ,

But the premise of all this ：

1. To accurately know the position relationship between two cameras , That is, the rotation of one camera to the other $R$ Move peacefully $t$, namely The external parameters of the camera
2. To accurately know the internal equipment parameters of dual cameras , For example, like the main point $(x_0,y_0)$、 The focal length $f$、 Distortion parameters, etc , namely The internal parameters of the camera

Introduction to internal and external parameters , Please see the previous article ： Introduction to stereo vision （1）： Coordinate system and camera parameters

Dual camera calibration

In general , We have seen the most binocular stereoscopic , Fixed on a rigid structure by two different cameras , Form a stable binocular stereo system . In use , There will be no obvious mutual movement between cameras , The stable dual view structure can complete the acquisition of single frame depth information at any time . The following figure shows some examples ：

We said earlier , The camera internal parameter matrix can be obtained by a single camera $K$、 External parameter matrix $R,t$, Compared with single camera calibration, dual camera calibration

1. One more relative external parameter , That is, the rotation matrix of the right camera relative to the left camera $R$ Vector of peaceful shift $t$.
2. Two absolute external parameters are missing , That is, the external parameter matrix calibrated by the single camera itself is no longer needed , Just keep 1 The relative external parameters obtained , It is called the external parameter of stereo vision system .（ In fact, absolute external parameters are generally not required for single camera calibration , It is only natural output as calibrated output , In dual camera calibration , Absolute external parameters can be used as intermediate variables to calculate relative external parameters ）

We said at the beginning of the article , Dual camera calibration is based on single camera calibration , I think smart students have come up with some ideas , First, we use the same shooting method to collect the calibration pattern , Most of the operation methods and precautions are the same , See Introduction to stereo vision （3）： Zhang calibration method of camera calibration 【 Super detailed, worth collecting 】

The difference is this ：

1. Every time you shoot , Pay attention to whether the overlapping area of two cameras is enough , Generally more than 1/2 The image area of , Of course, the bigger the better , At some relatively large angles , A certain degree of overlap can be sacrificed .
2. Note that the two cameras try to keep the same distance to collect , This is to ensure that the scale and clarity of the two views are consistent , Reduce the error caused by inconsistency .

Sketch Map ：

The algorithm part

After image acquisition , Enter the algorithm section , As mentioned earlier , The first step is to calibrate the two cameras individually , Get the internal parameter matrix of each of the two cameras K、 Absolute external reference $R,t$（ That is, the external parameters of the camera relative to the calibration plate at each acquisition ）, And distortion coefficient $d$（ there $d$ Is the set of all distortion coefficients ）
1. Do single camera calibration for two cameras separately , Get the internal parameter matrix of each of the two cameras K、 Absolute external reference $R,t$, And distortion coefficient $d$

Then some students may think of , According to the absolute external parameters collected each time , That is, the relative external parameters can be calculated , But obviously we will generate N Relative external parameters （N Is the number of image pairs collected , That is, the number of acquisition positions ）, Which one is better ？ Take the mean ？ Or take the median ？

Experienced students will find out , Neither of these is reliable , Taking the mean value first has no robustness to significant outliers , As long as the absolute external parameter of a pair is wrong , The average value has a great influence ; Second, take the median because only one pair , The error average characteristic of multiple measurements is not used .

So the more reliable way is , Calculate the relative external parameters of each acquisition , And take the median value as the initial value , Based on minimizing the re projection error , Nonlinear iterative optimization obtains the optimal solution . Under this scheme , The internal parameters of the camera can be used as fixed values , The unknown number is N Set the absolute external parameters of the left camera $(R_i,t_i),i=1,..n$、1 Group right relative relative relative external parameters relative to left camera $R^{\prime },t^{\prime }$. The equation of minimum re projection error is the same as that of single camera calibration , Only the absolute external parameters of the right camera are calculated by the absolute external parameters and relative external parameters of the left camera .

2. Calculate the relative external parameters of each acquisition , And take the median value as the initial value , Based on minimizing the re projection error , Nonlinear iterative optimization obtains the optimal solution

Because the second iteration optimization , Camera internal parameters are fixed , The number of external parameters is also reduced, and there is a more accurate initial value , So iteration will be relatively easier to converge and more accurate .

summary

Stereo vision is the premise of calculating depth through stereo matching ：

1. To accurately know the position relationship between cameras , That is, the rotation of one camera to the other $R$ Move peacefully $t$, namely The external parameters of the camera
2. Know the equipment parameters inside the camera accurately , For example, like the main point $(x_0,y_0)$、 The focal length $f$、 Distortion parameters, etc , namely The internal parameters of the camera

Compared with single camera calibration, dual camera calibration ：

1. One more relative external parameter , That is, the rotation matrix of the right camera relative to the left camera $R$ Vector of peaceful shift $t$.
2. Two absolute external parameters are missing , That is, the external parameter matrix calibrated by the single camera itself is no longer needed , Just keep 1 The relative external parameters obtained , It is called the external parameter of stereo vision system .

The difference between dual camera image acquisition and single camera image acquisition lies in ：

1. Every time you shoot , Pay attention to whether the overlapping area of two cameras is enough , Generally more than 1/2 The image area of , Of course, the bigger the better , At some relatively large angles , A certain degree of overlap can be sacrificed .
2. Note that the two cameras try to keep the same distance to collect , This is to ensure that the scale and clarity of the two views are consistent , Reduce the error caused by inconsistency .

Steps of dual camera calibration algorithm ：

1. Do single camera calibration for two cameras separately , Get the internal parameter matrix of each of the two cameras K、 Absolute external reference $R,t$, And distortion coefficient $d$
2. Calculate the relative external parameters of each acquisition , And take the median value as the initial value , Based on minimizing the re projection error , Nonlinear iterative optimization obtains the optimal solution

So you

Homework

Here are some exercises for you , We can deepen our understanding through practice ：

1 adopt opencv The interface provided by the open source library completes the calibration of dual cameras .
2 Higher order is , You can be independent opencv Library to write a set of dual camera calibration algorithm ？ Or just use opencv To detect the corner coordinates , Other steps are implemented by yourself .

Please refer to the address ：https://github.com/ethan-li-coding/StereoV3DCode [ Sorry to let it go , The code has not been updated for a long time , But do you believe that I will be more right one day ？]