Multi camera data collection based on Kinect azure (III)

be based on Kinect Azure Multi camera data acquisition （ 3、 ... and ）

     Based on Kinect Azure Multi camera data acquisition （ One ） and be based on Kinect Azure Multi camera data acquisition （ Two ） in , We summarized the use of Kinect Azure The method of collecting 3D color point cloud by camera and the synchronization method between cameras . If camera synchronization is the basis of multi camera acquisition system , Then the point cloud registration part is the key part of the system . Final acquisition quality , All rely on accurate point cloud registration results to ensure . therefore , This article continues to summarize the next part of the acquisition system —— Point cloud registration . If there is an incorrect or imprecise part , Welcome all bloggers to make corrections .

One 、 Point cloud registration

     What is point cloud registration ？ Popular speaking , Is to move all point clouds in different coordinate systems to the same coordinate system . For multiple cameras in different positions that have been synchronized , Each camera will get a three-dimensional image at different angles , In order to synthesize point clouds from different angles collected by multiple cameras into a complete point cloud , At this point, point cloud registration is required .
     So how to perform point cloud registration ？ First, let's think about how to move the point clouds collected by two cameras to the same coordinate system . For the main equipment , The point cloud coordinates collected by it are the coordinates under the coordinate system of its depth sensor , For slave devices , The coordinates of the point cloud collected by it are also the coordinates under the coordinate system of its depth sensor . At this time, what we need to do is to find the conversion relationship between the depth sensor coordinate system of the slave device and the depth sensor coordinate system of the master device , That is, the external parameters between the two coordinate systems . The external parameters between the depth sensor and the color sensor of each device are known , Then we only need to get the external parameters between the coordinate systems of the two equipment color sensors , You can move point clouds in different coordinate systems to the same coordinate system , That is to achieve point cloud registration . So how to get the external parameters between the color sensors of two devices ？ There are two main methods ： Camera calibration and coarse matching + Fine matching .

Two 、 Camera calibration

     At present, the most commonly used calibration method is Zhang Zhengyou calibration method （ Black and white checkerboard calibration ）. For a single sensor , This method can calculate the internal parameters of the sensor 、 Distortion parameters and external parameters between sensor coordinate system and world coordinate system . For both sensors , External parameters between them can be calculated . This section first summarizes the principle of Zhang Zhengyou's calibration method , Then explain how to pass opencv Library to implement .

1、 Principle of Zhang Zhengyou calibration method

     Through the collected corners of the chessboard in the coordinates of the pixel coordinate system and the world coordinate system , The following relationship can be established （ The transformation relationship between coordinate systems is Kinect Azure Multi camera data acquisition （ One ） Has been described in ）：
$$sm=K[R,t]M,MakeH=\lambda K[R,t],bem=HM$$ among ,s Is the scale factor ,m Is the pixel coordinate system coordinates ,M Is the world coordinate system （ The chessboard plane is Z=0）.H by 8 Homography matrix of degrees of freedom , Theoretically, it can be solved by four pairs of matching corners .
     because $$H=[h_1,h_2,h_3]=\lambda K[r_1,r_2,t]$$$$r_1=\lambda K^{-1}{h}_1,r_2=\lambda K^{-1}{h}_2,t=\lambda K^{-1}{h}_3$$     And R It has orthogonality and normalization , therefore $$r_1\cdot r_2=0\Rightarrow h_1^T{K}^{-T}{K}^{-1}{h}_2=0$$$$\Vert r_1\Vert =\Vert r_2\Vert \Rightarrow h_1^T{K}^{-T}{K}^{-1}{h}_1=h_2^T{K}^{-T}{K}^{-1}{h}_2$$     One H A matrix can provide two equations , about 5 Degree of freedom K, At least three pictures are needed to solve . But the above derivation results are based on the solution in the ideal case , Because there may be Gaussian noise , Therefore, the maximum likelihood estimation is used for optimization . Use more pictures （20 Valid pictures ）, Each picture has more corners （8*6 Chessboard ）.
     According to the above principle, the external parameters of a single sensor can be obtained , The external parameters of the color sensors of the two devices are obtained respectively , The external parameters between the two color sensors can be calculated .

2、 Camera calibration is realized

     Can pass opencv Library to achieve camera calibration , Import header file calib3d.hpp. Use opencv The steps of camera calibration are as follows .

• The camera acquires a frame of image , From image format to Mat matrix
• utilize findChessboardCorners() Function to identify the position of checkerboard corner in an image
• drawChessboardCorners() Function to check whether the identified corner position is accurate
• When there are enough pictures , call calibrateCamera() To obtain distortion coefficient of the single camera ; call stereoCalibrate() To obtain the external parameters between the two cameras （R、t）

3、 ... and 、 Rough match + Fine matching

     Point cloud registration can also be achieved through rough matching + The method of fine matching . Rough match , For both devices , Find the feature points on the collected images , And match the feature points , recycling SLAM The method of antipolar geometry in , Find out a set of rough external parameter solutions . Fine matching , utilize icp Method iterative solution , The solution obtained by rough matching calculation is taken as icp The initial solution of the algorithm , call pcl library , It can be done by icp The algorithm calculates the exact external parameters .

1、 Rough match

     Feature point matching algorithm can be used , First, find the matching feature points on the image . Re pass SLAM Find a rough solution to polar geometry .
     Feature point matching algorithm can be selected sift Algorithm or orb Algorithm .sift（ Scale invariant feature transformation ） Algorithm , seeing the name of a thing one thinks of its function , It will have some special points , Not because of the change of perspective 、 Changes in light 、 The point at which noise disappears , As feature points . and orb（Oriented Fast and Rotated Brief ） Algorithm , It is through Fast Algorithm to find feature points , Find out what makes you stand out , Take a point and compare it with the points around it , If it is different from most of the points, it can be considered as a feature point . therefore ,orb The algorithm extracts feature points faster , but sift The robustness of the algorithm is better .
     Can pass opencv Library implementation orb/sift Algorithm , Header files need to be imported features2d.hpp.

	orb	sift
Create objects	Ptr orb = ORB::create();	SiftFeatureDetector siftDetect; SiftDescriptorExtractor descriptor;
Extracting feature points	orb->detect();	siftDetect.detect();
Computational descriptors	orb->compute();	descriptor.compute();
Create objects for matching	BFMatcher matcher();	Ptr matcher = DescriptorMatcher::create();
Matching feature points	matcher.match();	matcher->match();

2、SLAM Polar geometry

     After the above rough matching process , You can get a series of matching pixels , It can be obtained by the method of antipolar geometry R、t A set of solutions . use $p_1,p_2$ To represent two sets of matching pixel coordinates , use $P$ Express $p_1$ Coordinates under the corresponding camera coordinate system , Then there is the following relation ：$$s_1{p}_1=KP,s_2{p}_2=K(RP+t)$$$$x_1=s_1{K}^{-1}{p}_1,x_2=s_2{K}^{-1}{p}_2,x_2=R{x}_1+t$$$$t^{\wedge }{x}_2=t^{\wedge }R{x}_1+t^{\wedge }t=t^{\wedge }R{x}_1$$$$x_2^T{t}^{\wedge }R{x}_1=x_2^T{t}^{\wedge }{x}_2=0$$$$p_2^T{K}^{-T}{t}^{\wedge }R{K}^{-1}{p}_1=0$$$$E=t^{\wedge }R(BenqualityMomentfront),F=K^{-T}E{K}^{-1}(The\; baseFoundationMomentfront)$$

• E/F by 8 Degree of freedom matrix , Each pair of matching points provides an equation , At least 8 Points can be solved E/F. When there are too many matching points, we can solve the overdetermined equation E/F
• E The singular value of satisfies $[\sigma ,\sigma ,0]$,σ Is the scale factor , Can be calculated $E{E}^T$ To prove that
• Yes E Conduct SVD decompose ,$E=U\Sigma V^T=\left(U{\Sigma }_1{U}^T\right)\ast \left(U{\Sigma }_2{V}^T\right),\Sigma ={\Sigma }_1{\Sigma }_2$, The former is an antisymmetric matrix （t^）, The latter is an orthogonal matrix （R）
• Due to the existence of scale factor , Found t There may be positive and negative ; Due to different rotation directions ,R There are also two situations . So there are four sets of solutions , But only one set of solutions satisfies P The depth is positive in both coordinate systems , The depth can be calculated by triangulation
• Triangulation ：
  $$s_1{p}_1=KP,s_2{p}_2=K(RP+t)$$$$s_2{K}^{-1}{p}_2=R\left(s_1{K}^{-1}{p}_1+t\right)$$$$p_2^{\wedge }KR\left(s_1{K}^{-1}{p}_1+t\right)=s_2{p}_2^{\wedge }K{K}^{-1}{p}_2=\overrightarrow{0}$$$$s_1{p}_2^{\wedge }KR{K}^{-1}{p}_1=-p_2^{\wedge }KRt$$
  Find out s1 after , Carry in and calculate P, And then find out s2, if s1 and s2 All are greater than 0, Then the group solution is correct R、t.
• Can also pass opencv Library findEssentialMat/findFundamentalMat Function solving E/F matrix , adopt recoverPose Function solving R、t

3、 Fine matching

     utilize ICP（point2point,point2plane etc. ） Algorithm , The external parameters obtained by rough matching are used as ICP The initial solution of the algorithm , call PCL library , Realization ICP Algorithm , So as to obtain the external parameters .
     Through the above camera calibration or rough matching + The method of fine matching , External parameters between color sensors can be obtained , Thus, the external parameters between the two equipment depth sensors are calculated , And through external parameters , Move the two point cloud data to the same coordinate system , Compose a complete 3D image .
     thus , The whole system is set up , Review the whole process of the system again . First, synchronize multiple cameras , Make the data they get at the same time . then , Each camera collects and processes data separately . Last , Through point cloud registration, 3D images from different angles collected by multiple devices , Move to the same coordinate system , Synthesize a complete point cloud data .
     After the acquisition system is completed , You can continue to build a complete collection 、 Compress 、 transmission 、 Display system . adopt windows Use of multithreading , Collect data 、 Data processing 、 Data transmission is divided into multiple threads , The collected data can be displayed in the client in real time .