3D reconstruction - stereo correction

Click on the above “ Computer vision workshop ”, choice “ Star standard ”

The dry goods arrive at the first time

source ： official account @ Computational photography

One . Intuitive understanding of stereo correction

In the article 68. Three dimensional reconstruction 3- Two view geometry in , We see through triangulation , It can determine the position of an image point in three-dimensional space , The premise is that we get the corresponding point of this image point in another image in advance , And I know the camera matrix of the two cameras .

Get paired matching points in two images , The same 3D The projection point of the point on two images is very important . In the above article, I showed you the diagonal geometric constraints in the two view geometric model , So when you need to search like dot x Corresponding matching point of x' when , We can control the search scope to the polar line of the second image l' On .

However ,l' Usually not horizontal , So we need to search on an oblique line segment of the second image . Such an algorithm is very inefficient , Especially when we need to use the features of image blocks to make point matching judgments . In later articles , I will introduce ” Stereo matching “, This process needs to find matching points for all points in the image , If each point is still operated according to this method of oblique line segment search , It will be very, very slow . therefore , We must introduce stereo correction (Rectification) operation , It transforms two images , Finally, the polar lines of the two images are parallel , And align horizontally . such , When we need to search the corresponding matching points , You only need to do one-dimensional search in the horizontal direction , Greatly accelerated the speed .

This process , It's like we created two virtual cameras with the same internal parameters , They point in the same direction to shoot the original scene , Get two new images , As shown in the figure below ：

The following shows a pair of images taken with two cameras of a dual camera phone in the same scene , The two cameras have different internal parameters , Therefore, there are obvious spatial transformations and scale differences between the perspectives of the captured images .

The camera of this mobile phone is arranged vertically , The captured image is shown in the following figure

So when we draw some polar lines , You will find that these polar lines are inclined , No alignment .

Even if we scale the image in a certain proportion according to the internal parameters , Make the two images the same size , We found that the polar lines still couldn't be aligned , And it is inclined ：

After performing the stereo correction steps I mentioned above , All epipolar lines of both images will become horizontal , And align vertically , At this time, the scale of the two images is also corrected to the same state . Note that these two cameras are arranged vertically on the mobile phone , In order to make the polar line in the horizontal direction , I made the image 90 Degree of rotation . If you don't do this rotation , The polar lines are aligned vertically , This does not use the processing of subsequent stereo matching algorithms .

Just for convenience , We can adjust the image polar line to vertical , Observe and compare ：

Two . Principle of stereo correction algorithm

2.1 Theoretical derivation

Officially , Image stereo correction refers to a plane projection transformation of two images respectively , Make the corresponding polar lines of the two images in the same horizontal upward , And the opposite pole is mapped to infinity , In this way, there is only parallax in the horizontal direction between the two images , Stereo matching problem from two-dimensional Down to one dimension , Thus, the matching speed is improved .

Although there are various stereo correction algorithms , Today I will only introduce a very easy to understand method , And it is also a widely used method ：

The author gives a simple method of image correction , You only need to know the camera matrix of two cameras , And the angle of view of the two cameras is not much different —— This assumption is true in the projects I face , So I prefer this algorithm .

Now let's do a little simple derivation , Please hold it firmly

By article 66. Three dimensional reconstruction 1—— We know the geometric model and projection matrix of the camera

And because of

And from Xw The straight line to the projection imaging point must pass the optical center , As shown in the figure below

therefore , We can express the coordinates of three-dimensional space points as the following formula ( The upper horizontal line represents homogeneous coordinates ),c It refers to the coordinates of the optical center ,\lambda It's a scale factor , It's used to regulate Xw The position of is on this line .

Next , Let's take a look at the geometric models of the two cameras before and after correction ：

so , Before correction and after correction , The projection matrix of the camera has changed , So the same 3D The projection point of the point has also changed . After correction , We see that the principal planes of the two optical centers are parallel , They all look in the same direction . We use subscripts o representative origin, Use subscript n representative new, that , The above formula becomes ：

In stereo correction , We know the camera matrix of two cameras , Therefore, we know the optical center coordinates of the two cameras . meanwhile , Before and after stereo correction , Now let's talk about Fusiello The algorithm of et al assumes that the optical center is constant , So we can combine the above two formulas , Get the relationship of projection points ：

among ：

therefore , In order to calibrate the two cameras , We only need to determine the internal parameters and rotation matrix of the new virtual camera .

Let's start with internal parameters ： After correction , We need two virtual cameras with exactly the same internal parameters , So in Fusiello And so on , Directly use the following formula to get the internal parameters of the new camera ：

Look at the rotation matrix ： After correction , The direction of the two cameras is the same , Their rotation matrix can be expanded as follows ：

among , The three rows of the rotation matrix represent the new camera matrix X/Y/Z Axis , They have the following characteristics respectively ：

• 1. X The axis should be parallel to the line connecting the optical centers of the two cameras , So there is ：

• 2. Y The axis should be perpendicular to X Axis and line of sight direction of the original camera ( That is, of the original camera coordinate system Z Axis , We use it k To express ) The plane where it is ：

• 3. Z The axis should be perpendicular to X Axis and Y The plane of the axis ：

Only this and nothing more , We determined the direction of the virtual camera （ Rotation matrix ）, Light heart （c1 and c2), Internal reference , In this way, it is easy to use the formula just mentioned to transform the image ：

2.2 Concrete realization

In practice , We assume that it has been calibrated by the geometric parameters of the camera （ See article 67. Three dimensional reconstruction 2—— Camera geometric parameter calibration ） Internal and external parameters of two cameras are obtained , And the image has been corrected for distortion . that , The whole calibration process can be summarized as follows ：

In the author's paper , Lists 22 That's ok Matlab The code completes this process , You can check . The author also put the relevant code 、 The data is in sci.univr.it/∼fusiello/rect.html For us to learn , nice ！

The key points of this algorithm are as follows ：

• The algorithm itself does not consider the lens distortion of the image , Therefore, it is necessary to correct the distortion of image points before using

• Pay attention to the layout between the original cameras . As described at the beginning , If the cameras are arranged vertically , You need to rotate the image to the horizontal first , Then the horizontal polar alignment can be carried out . such as , The layout of common mobile phone multi cameras is as follows , When applying the above algorithm , You need to rotate the image in advance , In order to ensure that the polar line after correction is in the horizontal direction

• The whole algorithm relies heavily on accurate internal parameters 、 External reference . If there is any error in this information , The polar lines after final correction may not be aligned horizontally .

2.3 Effect display

Here I show you more images taken with the same mobile phone , You can clearly see the change of the epipolar line of the image before and after correction ：

Before correction ：

After correction ：

Before correction ：

After correction ：

3、 ... and . summary

Today I introduce you a very simple , Easy to reproduce stereo correction algorithm . in fact , My actual project is also extended on the basis of this algorithm , And then carry out practical application . So I think this algorithm is very practical , To use , I recommend you to study , I hope I can give you some inspiration .

If you find that the epipolar line cannot be aligned horizontally in actual use , It is necessary to consider the accuracy of calibration . For example, there is a slight change between the state of calibration and the state when you actually use the camera —— This is very common —— You need to consider how to further tune the relevant parameters , In order to better use this algorithm .

Four . reference

• CMU 2021 Fall Computational Photography Course 15-463, Lecture 17

• Andrea Fusiello, et al, A compact algorithm for rectification of stereo pairs

• Richard Szeliski, Computer Vision: Algorithms and Application.

• Stefano Mattoccia, Stereo vision: algorithms and applications, Università di Firenze, May 2012

• 68. Three dimensional reconstruction 3—— Two view geometry

• 67. Three dimensional reconstruction 2—— Camera geometric parameter calibration

This article is only for academic sharing , If there is any infringement , Please contact to delete .

blockbuster ！ Computer vision workshop - Study Communication group Established

Scan the code to add a little assistant wechat , You can apply to join 3D Visual workshop - Academic paper writing and contribution   WeChat ac group , Aimed at Communication Summit 、 Top issue 、SCI、EI And so on .

meanwhile You can also apply to join our subdivided direction communication group , At present, there are mainly ORB-SLAM Series source code learning 、3D Vision 、CV& Deep learning 、SLAM、 Three dimensional reconstruction 、 Point cloud post processing 、 Autopilot 、CV introduction 、 Three dimensional measurement 、VR/AR、3D Face recognition 、 Medical imaging 、 defect detection 、 Pedestrian recognition 、 Target tracking 、 Visual products landing 、 The visual contest 、 License plate recognition 、 Hardware selection 、 Depth estimation 、 Academic exchange 、 Job exchange Wait for wechat group , Please scan the following micro signal clustering , remarks ：” Research direction + School / company + nickname “, for example ：”3D Vision  + Shanghai Jiaotong University + quietly “. Please note... According to the format , Otherwise, it will not pass . After successful addition, relevant wechat groups will be invited according to the research direction . Original contribution Please also contact .

▲ Long press and add wechat group or contribute

▲ The official account of long click attention

3D Vision goes from entry to mastery of knowledge ： in the light of 3D In the field of vision Video Course cheng （ 3D reconstruction series 、 3D point cloud series 、 Structured light series 、 Hand eye calibration 、 Camera calibration 、 laser / Vision SLAM、 Autopilot, etc ）、 Summary of knowledge points 、 Introduction advanced learning route 、 newest paper Share 、 Question answer Carry out deep cultivation in five aspects , There are also algorithm engineers from various large factories to provide technical guidance . meanwhile , The planet will be jointly released by well-known enterprises 3D Vision related algorithm development positions and project docking information , Create a set of technology and employment as one of the iron fans gathering area , near 4000 Planet members create better AI The world is making progress together , Knowledge planet portal ：

Study 3D Visual core technology , Scan to see the introduction ,3 Unconditional refund within days

  There are high quality tutorial materials in the circle 、 Can answer questions 、 Help you solve problems efficiently

Feel useful , Please give me a compliment ~