CVPR 2022 | Virtual Correspondence: Humans as a Cue for Extreme-View Geometry

Thesis link ：http://people.csail.mit.edu/weichium/virtual-correspondence/top.pdf

3D Reconstruction is a very classic problem in graphics , do 3D In the process of reconstruction , Often use multi view geometry , That is, for the same scene （ object ）, Observe from different perspectives , Then according to the observation The common part , utilize parallax Carry out three-dimensional information estimation . An important factor affecting the reconstruction effect is the need for different pictures The common part More , Otherwise, the reconstruction effect will be very poor .

however , For the above two figures , Although their shooting angles are almost different 180°, And there are differences in time , However, it is easy to judge from human observation that these two diagrams represent almost the same scene . Why is that ？ Because people will also consider the characters in the picture Posture 、 Appearance （ size ）、 identity And so on , Unlike multi view geometry, which only uses feature point information .

Based on this , This paper presents a method , Semantic matching can be performed on two graphs with few common perspectives , This method is based on analyzing the corresponding relationship of people in the scene , And can restore the camera pose in each picture .

The author first raises a question ： Is it necessary to restore the pose of the camera with the corresponding three-dimensional points on different pictures ？ The answer is No , The traditional polar geometry requires that the observed points are the intersections of the polar lines , Here Only the polar lines passing through the points intersect, that is, the two points are considered to be corresponding （ This article is called Virtual Correspondence, abbreviation VC）.

however , To judge whether two polar lines intersect , Generally, you need to know the pose of the camera , This is a dead circle , That is, now I want to use the polar line that intersects two points to restore the camera pose , However, the determination of the disjoint of polar lines depends on the camera pose .

therefore , The author's idea here is , Using prior knowledge —— people , Because the human model has strong prior knowledge , There has been a lot of work to restore the human posture model according to a single picture , And with mannequins , It's easy to judge that the polar thread passes through two people “ Point of intersection ” Where it will appear , Thus, the points on different perspectives can be matched （ For example, the ray passing through the chest can quickly find the corresponding point on the back ）.

meanwhile , Although here VC It is different from the traditional point by point matching , But it's easy to put classic SfM(Structure from Motion) Method to modify , Used to restore the camera pose .

Selling points of this article （ contribution ）：

1. Put forward VC, Based on the traditional epipolar geometry , Better applicability .
2. A human model is proposed to estimate VC Methods , And it can be compared with the existing 3D frame （ Such as SfM） Good compatibility , It has a wide range of applicable scenarios .
3. This method can be combined with some downstream tasks （ Such as multi view geometric reconstruction , Any perspective generation ）

The above figure is the flow chart of the algorithm , First, recover human's from the picture 3D Model , Then randomly emit rays , Record all the collision points between it and the model （ Such as the lower abdomen on the front and the back on the back ）, Then find their corresponding pixels in the two images , So I found VC.

With VC after , The next step is to estimate the camera pose through these corresponding relationships . Practice and SfM similar , Just replace the traditional matching feature points with VC. meanwhile , because VC Point comparison depends on the accuracy of human shape estimation , So there will be some noise, The method is to reduce the influence of error by optimizing the re projection error as a whole （ I only know some basic knowledge in this part , Interested readers can study deeply by themselves ）.