[2021]IBRNet: Learning Multi-View Image-Based Rendering Qianqian

NeRF A big problem is that it can only represent one scene , Therefore, this article proposes a framework for learning multiple scenarios at the same time , And it can be extended to scenes that have not been studied .

This article and NeRF The biggest difference is that the input data does not only have a target perspective , There are also corresponding multi view pictures of the same scene , Therefore, theoretically, it can be directly applied to new scenarios end-to-end .

Model flow ：

1. Input the multi view pictures of the same scene into the network （ There is no limit to the number of ）, Then use a U-Net To extract each picture （source view） Characteristics of , Features include image color , Camera parameters , Image representation （ Here it can be understood as NeRF Emitting light to the radiation field in , Then save the corresponding light parameters and image features ）.

2. Then input the features of each picture in parallel transformer, Used to predict a common color and density . The reason for the common color and density is that the features input from multiple perspectives are the features of the same point in different perspectives by default , So the result is to predict our target perspective （target view） Results at this point .

3. Render the result by volume rendering , Then optimize the network through the reconstruction loss of pixels

4. Another scene , repeat 1~3

remarks ： If you keep training with the same scene , In theory, the effect will be better , That's what the paper mentioned finetune The situation of .

Personal understanding ： In essence, the model here learns “ How to interpolate ”, Instead of building a radiation field , Therefore, it may not perform well in sparse situations or complex scenes