Use 4D nerf to display occlusion (cvpr2022)

title ：Revealing Occlusions with 4D Neural Fields

source ：CVPR 2022 (Oral Presentation)

Reading & translate ： Bamboo stone

article & Code & Video lectures ：https://occlusions.cs.columbia.edu/

Occlusion is the basis of computer vision , Predicting the content behind occlusion is the basis of many applications in video analysis . In recent years , Researchers combine modern deep learning features with various methods to track occlusion . These include classical Kalman filter or linear extrapolation[25]、2D Recursive neural network [57] And a clearer reasoning mechanism [49]. This article combines cloud modeling Point cloud model 、neural field Neural field and attention mechanisms Attention mechanism to solve this problem .

This paper introduces a method for learning from RGB-D Camera prediction 4D The architecture of point cloud . The key of this method is the continuous neural field representation of point cloud , It uses the attention mechanism to regulate the observation of the whole space . Because the representation is continuous , This method can learn to generate points at any position in space-time , This allows high fidelity reconstruction of complex scenes . In the case of occlusion and missing scene structure , This representation can use attention to find the object and / Or missing scene structure , And then correct .

The main contributions of the articles are three ：

• Proposed from monocular RGB-D( In depth information ) Video input 4D Dynamic scene complements this new basic task , It lays a foundation for spatiotemporal reasoning tasks