Camera calibration (2): summary of monocular camera calibration

1. Review of coordinate system transformation

• First, the world coordinate system is rotated 、 The translation matrix is transformed into the camera coordinate system
• The camera coordinate system passes through similar triangles （ Principle of pinhole imaging ）, Convert camera coordinate system to image coordinate system , What we use here is actually the focal length of the camera .
• It is best to use camera internal reference , Convert image coordinate system to pixel coordinate system .

Camera calibration

The purpose of camera calibration is to obtain the internal and external parameters and distortion parameters of the camera .

Camera calibration process

Calibration process

• 1. Print calibration board
• 2. Shoot the calibration plate from different angles
• 3. use opencv The characteristic points in the interface detection calibration board （ Generally, square is used , Check its corners ）
• 4. Find the internal and external parameters of the ideal without distortion
• 5. The maximum likelihood estimation is used to iteratively optimize the accuracy （ Use Zhang Zhengyou calibration method to solve internal and external parameters , In fact, it is an initial value of iterative solution of maximum likelihood estimation , Given the initial solution parameters , In order to obtain more accurate internal and external parameters, we need to use iterative method to optimize ）
• 6. The radial distortion parameters are solved by the least square method
• 7. Comprehensive internal and external parameters 、 Distortion parameters , Using the maximum likelihood method , Improve estimation accuracy
• 8 It's best for us to calculate internal and external parameters distortion coefficient of the camera

Camera calibration trick

Calibration plate pattern
We usually use checkerboard to calibrate the board pattern , But we don't have to use checkerboards , Other patterns can be used .

• have access to The three dimensional 、 A two-dimensional , circular , grid , Random patterns .（ If you use black and white checkerboard , It will be easier to detect corners ; If you use a circle , It will detect the center of the circle .Opencv Both interfaces are provided ）
• Generally choose to use multiple perspectives of plane objects , Instead of constructing three-dimensional objects .

Number of checkerboard corners and parameters

• The number of calibration plates is generally the more the better , There are more grids per calibration board . Without introducing noise
• Total number of parameters ： Four camera internal parameters ($f_x,f_y,u_0,v_0$),5 Distortion parameters （k1~k3,p1,p2）,3 Two rotation parameters ( One parameter per axis , Can determine the rotation matrix ),3 Translation parameters .
• Suppose the checkerboard has N Corner points ,K Images in different positions
• K Checkerboard provides 2NK A constraint , Two constraints per corner
• Temporarily ignore the distortion parameter ： Yes 4 Internal parameters ,6 External parameters ( Take different pictures in different positions , The rotation translation matrix is different , The external parameters are different )
• The premise of being able to solve parameters : 2NK( constraint ) >6K +4, Introduction (N-3)*K>2 , Only when the number of equations is greater than the parameters can it be solved
• Homography matrix , adopt 4 A point can be uniquely determined ,4 A point can express a plane perspective , stay 4 Extend edges in both directions
• No matter how many points there are on the chessboard , Only four effective corner information are actually useful
• K>1 , You need to shoot at least two checkerboards
• Considering noise and numerical stability , You need to use a larger checkerboard , Capture more images
• For more effective effect , In general use 10 More than one ,7*8 Or a larger chessboard . Collect more pictures , Filter these graphs , Choose a more effective picture , Images with high noise can be removed .

How to evaluate the calibration effect of the camera

Evaluate the re projection error ：
Without the real parameters of the camera , Evaluate the effect of camera calibration

• Corner of the detected two-dimensional image , The distance between the projection point and the real-world three-dimensional point ;
• Use the camera internal and external parameters obtained by calibration , Project the three-dimensional world points to the two-dimensional points in the pixel coordinate system
• And then with the corner of the two-dimensional picture detected by the algorithm , Calculate the mean square error RM
  We will take a camera to shoot a checkerboard , Shooting a checkerboard will get a two-dimensional image , Two dimensional images can be used opencv Feature point detection algorithm , It can detect the corner coordinates of the two-dimensional image A; in addition , When shooting a checkerboard, we know its coordinates in the world coordinate system , Then we use the camera internal and external parameters and distortion parameters we have solved , To coordinate matrix transformation , The coordinates of the corners of the chessboard are obtained through calculation B. And then to solve A,B The mean square error of RMS
  
• It is acceptable that the average projection error is less than one pixel
• Calculate the re projection error of each graph , It can be used to select pictures with large errors
  
  External parameter visualization
• View of the calibration board with the camera as the center
• View of the camera centered on the calibration plate , Visualize external parameters
• The camera is fixed , Calibration plate movement , Suitable for camera centric views
• conversely , It is suitable to use the view with calibration as the center
  

Observe the image after distortion correction
- Observe the chessboard after distortion correction , Whether the curved line becomes straight , It is suitable for cases with obvious distortion

How to improve the effect of camera calibration

• Add more pictures

• The calibration plate covers the complete image , Especially the image edge 、 Where the distortion is relatively large at the corner

• The calibrated image needs enough changes , Different angles , Different positions

• Remove picture , For noisy pictures

• Remove blurred pictures

• The inclination of the removed calibration plate relative to the camera is greater than 45 degree