[2021]NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections

NeRF A big problem is that the requirements for the photos taken are relatively high , For example, the illumination should be consistent , There should be no moving objects around . However, in reality, the large number of multi view images we can obtain usually cannot meet these two points . For example, if we want to restore Tiananmen Square , But people usually come and go in front of Tiananmen Square , It's very difficult to take pictures without people . Therefore, this paper proposes a scheme , You can relax the light changes and instantaneous occlusion （ That is, moving shelter ） The limitation of .

Basic modeling and NeRF It's the same , I won't go into that , Let's talk directly about how he solves the problems of light change and instantaneous occlusion .

Light variation ：

First , This article borrows Generative Latent Optimization（GLO） Methods , I learned a set of pictures from each perspective embedding vector （appearance embedding）, then Put this embedding Added to the a priori of color , That is, perspective a priori .

Take a look back. NeRF, We are assuming that a ray of light is emitted from the observation point , Then irradiate into the radiation field , Then the density and color of the corresponding points on the ray path are obtained , Therefore, the only condition that can ultimately determine the color we get is our observation point , That is, the parameters of the camera . But in fact , Because the illumination in the multi view pictures we have is different , And different illumination will also make the color we get different . therefore , In order to take light conditions into consideration , The simplest way is to directly number the corresponding pictures （ That is, the corresponding image embedding vector ） Enter... As a condition NeRF, So that the observation light emitted by us can get different colors .

in addition , It is worth mentioning that , Density is not affected by light , So there is no need for pictures embedding Do a priori .

Instantaneous occlusion ：

First , The author will NeRF The radiation field in is divided into two overlapping radiation fields , One is static field （static）, That is, the field of the object we really want to restore , The other is instantaneous field （transient）, That is, those moving obstructions .

Treat static fields and primitives NeRF It doesn't make any difference , In the instantaneous field, the author requires that light not only give color and density , It also needs to be Give an uncertainty （uncertainty）, That is, how reliable is the current observation point . ideally , The model needs to be for pedestrians , Moving objects such as cars give very low confidence , And give more credibility to the static landscape .

in addition , Because those moving obstructions are different in every angle of view , therefore The author also adds a visual angle a priori to the instantaneous field （transiant embedding）（ It is essentially the same set of things as the priors in light changes , Just for different purposes ）.

The final optimization is the same as NeRF equally , Use the volume rendering result and the original image to do the reconstruction loss .