3D content generation based on nerf

source ： Deep Blue College 《 be based on NeRF 3D content generation 》 Zhang Kai

3D content

Generate from pictures 3D content 【 Labor takes time 、 energy 】 => Automatically generate by computer-aided means

picture ： It's very easy to get => Anti rendering Generate 3D content

Computer graphics ： How to generate high-quality rendered images
Computer vision ： Given picture => Anti rendering generates 3D content needed in computer graphics 【 Can change the light , Insert objects and so on 】

The three elements of anti rendering

1. Shape representation

Triangle Mesh、Point Cloud、Occupancy field、Signed distance filed
Different shape representations may determine the difficulty of solving the problem 【 Different optimization methods 】

2. appearance

The left side represents the material and light separately 【 Ideal situation , You can change the light , Edit material , But it's very difficult to understand （ It involves the rendering process in graphics ）】, On the right, pack the material and light together 【 Can't edit well , Put the object in a new environment and observe its appearance ,】

---

Anti rendering is relative to 3D reconstruction , The key is differentiable,2D->3D->2D
3D reconstruction is a special case of inverse rendering , 3D reconstruction was not focus Rendering quality .

---

3. Rendering process 【 Optimize 】

Ray tracing rendering: Track the propagation of light , Pass through each in the image pixel The process of weighted summation of light direction color distribution => Get the color of pixels

NeRF

Right picture ： The quality of depth map is very high

Soft Shape: Like fog , A little bit appears in every part of the space , Unlike solid objects, they only occupy a small part of space .

Success factors ：
1. Shape representation 【 Soft shape representation （ Foggy ）】
2. appearance 【 Materials and light 】
3. Rendering process 【 Functions are all differentiable 】


The geometric details on the right may not be good enough 【NeRF shortcoming 】

The previous work with neural network is not good enough => Hard representation is selected （eg. Triangle Mesh）

Reasons for the success of selecting soft shapes ：

shortcoming ： Every point in the light predict Color =>Expensive

=》 Prove the validity of soft shape , Better rendering can still be achieved without neural networks
There is no neural network （evaluation Very slowly ）, For every pixel of light, go querry

At the beginning, neural network is introduced to represent the scene ：
Another question ： Neural networks have special bias, Tends to fit smooth shape=> Introduced a map $Y$

Five scenarios ：

1. 360 No background , Only the prospect
2. Only move the camera within a small range
3. Panoramic shooting mode
4. Take photos casually with your mobile phone in your room 【 Cameras are more irregular 】
5. 360in and outforward Scene , Want to rebuild both the foreground and the background

Local or The overall There is one trade off (a) Choose the point on the foreground (b) Divide the points into the foreground and the background
Yes resolution problem

NeRF++

Draw a ball to deal with the foreground and background

such wrapping The property of can well solve the problem of resolution ： Space is squeezed




NeRF It has good composition The nature of

NeRF At present, it cannot run on real-time online devices , It can't support editing well .

NeRF The sampling frequency corresponds to pixel size
NeRF Core assumptions : Objects are static

The sawtooth problem occurs when downsampling , It is related to image sampling frequency .

Nyquist frequency problem