NeRF: The Secret of 3D Reconstruction Technology in the Popular Scientific Research Circle

关注公众号,发现CV技术之美

 

刘东生

-  Lenovo Morningstar Algorithm Engineer

-【Morning Star Classroom】讲师

What is the hottest 3D vision technology in the past two years?,相信NeRF是一个绝对绕不过去的名字.

Once this technology is proposed,就被众多研究者所重视,对NeRFscrambling to explore、研究、改进,It only took two years for the technology to become mainstream in the field of 3D vision.

这说明什么？我认为,这恰恰说明了NeRF拥有Super strong 3D expression ability and wide range of potential applications.

今天,我们就来看一看NeRF是什么,学习下该方法到底强在哪里？

NeRF到底是什么？

NeRF（Neural Radiance Fields神经辐射场）最早是在2020年ECCV会议上的最佳论文中提出的概念,其将隐式表达推上了一个新的高度,仅用2D的posed images作为监督,即可表示复杂的三维场景.

The task it has to deal with is new perspective synthesis.We use an intuitive expression to understand what it does,如下图所示,If we need a new Angle of view synthesis of excavator in this picture,First we will collect images from different angles around the excavator,Then calculate the camera pose for each acquisition angle,The collected image sequences and their corresponding poses are sent to theNeRF,some new perspectives.

That is, we look at this excavator from any position in space,what the image you see should look like,NeRFcan be synthesized.

相比传统,NeRF到底“香”在哪

Traditional 3D reconstruction can be divided into three types according to whether the input image is a depth map or a color map.主动式和被动式Two ways to rebuild,Their principle is shown in the following two figures：

被动式

In traditional passive 3D reconstruction,First take color images of the object to be reconstructed from different angles,随后通过SfM(structure from motion)and other techniques to obtain the camera pose and the initial point cloud of the model.Then through the depth estimation、Dense reconstruction of point clouds、Grid reconstruction and optimization and the grid mapping process to get the final model with texture.

主动式

In traditional active 3D reconstruction,First, take depth images of the object to be reconstructed from different angles,Since the color image helps with camera positioning and adding color to the model,Can also be in acquisition depth image at the same time,To collect the corresponding color images.随后通过ICP（Iterative Closest Point）and other techniques to calculate the camera pose.The scene is then implicitly expressed asSDFVoxel grid model,最后通过raycastingRender out the reconstructed perspective,最后输出给AR设备显示.

Traditional 3D reconstruction has many disadvantages,For example, there may be holes in the final reconstructed model、texture aliasing、Because the voxel resolution limit lost a lot of details.

而NeRFPhoto-level new perspectives can be composited,Reconstruction of the model more rich details,It optimizes the underlying continuous volumetric scene function by using a sparse set of input views,Achieves the best results for synthesizing complex scene views,无空洞、细节还原,And because of the large number of people in the research,development is particularly rapid,That's it“真香”所在.

图 | 来自instant ngp

NeRF是如何做的？

图片来自：原始版本Nerf

for a proxy object,首先我们会选定一个boundingboxSurround the object to be rendered,For a ray emitted from a view angle to a pixel in an image,We will sample many points on this ray,Each have a black dotxyzindicate its location.

the angle of observation.

MLPThe input to the network isboundingboxfive degrees of freedom for all black dots in.对于每个采样点MLPThe network will calculate the color and density of this point,Density can be understood as the point in this article on the direction of ray a visibility.The density of each point is onlyxyz有关,The color of each point is not only related to the position of the point, but also related to the viewing angle..之后通过volume renderingRender out the color of the pixel,Errors from the real color,to train the network.

NeRF也有缺点？！

虽然NeRF的优势显而易见,but in the original versionNeRFthere are many disadvantages：

（1）Both training and rendering are slow

（2）Can only represent static scenes

（3）经过训练的NeRFIndicates that it will not generalize to other scenarios.

针对NeRF的不足,In the past two years, there have been various works to continuously improve it.The main improvements are concentrated in four directions：

（1）For a single sceneNeRF,from the rendering quality、scene size start.Major improvementsMip-NeRF、NeRF++、Mip-NeRF-360、NeRF-W、Block-NeRF、Urban-NeRF、CityNeRF等.例如Block-NeRFHigh-quality reconstruction of a city-size scene.

图片来自：源自Block-NeRF

（2）Number of images for training：原始版NeRF需要100Zhang left and right pictures for training,而PixelNeRF、IBRNet、MVSNeRFSuch as working just a few images can be high quality reconstruction.

↓

图片来自：源自PixelNerf

（3）Speed ​​for training and rendering：原始版NeRFIt takes dozens of hours to train a small scene,而Plenoxels、instant-ngpEtc work greatly speeds up the training process,instant ngp甚至只需要5sto train a scene.原始版NeRFThe rate at which the image is rendered is only0.06fps,而最近的FastNeRF、SqueezeNeRFWait until the work can be200fpsrender at the speed of.

图片来自：instant NGP

（4）针对特定场景的NeRF的改进：Apart from the above mentioned,Such as specular reflection、针对人脸、For the dynamic human field,There is also a lot of improvement work.

图片来自：instant DNerf

Most importantly, where！

关于NeRF的应用,We naturally think of autonomous driving.The video above isNvidia在今年3at the developer conference inNvidia自动驾驶的Demo.用NeRFReconstruction can do real-time fusion virtual and reality.

下面的视频展示了NeRFCan simulate some complex road conditions,Such as rain and snow on roads humidifying,In fact, it provides a good simulation data for autonomous driving.

此外,像BlockNeRFwait for workRecreate realistic scenes at city level,It is also possible to perform lighting rendering on the scene under different conditions.可以用NeRFTechnology to build 3D models of entire cities,Digital map as a metaverse,Can also be used as a game map.

此外,For industrial scenarios,NeRF可以Rebuild detailed 3D models,用于工厂的AR巡检等任务.科研、工业、能源、制造......Create realistic 3D models and scenes in almost all fields,结合XR技术后,with the current hot“元宇宙”概念不谋而合,也许这就是NeRFThe reason for the increasing popularity！

一

x

欢迎提问

亲,Is the above content correct?NeRFThe concept and application of？其实这些只是“冰山一角”...

what else aboutNeRF的问题,Welcome to leave your questions in the comment area~

「文章来源」

公众号：联想上研 

文章作者：Lenovo Morningstar Algorithm Engineer 刘东生

END

欢迎加入「三维重建」交流群备注：3D