Brief analysis of ref nerf

elementary analysis Ref-NeRF

NeRF It is a popular 3D rendering network , This article is based on the new RefNeRF As an opportunity , Try to explain the close relationship between normals in 3D surfaces and rendering .

---

• Research background

  • rendering method
  • NeRF The problem of
• The new method

  • The main idea
  • How important are normals ？
  • Network structure
• result

  • And NeRF Comparison of methods
  • Scene editing

Research background

rendering method

We simplify the rendering process to ： When shooting an object , Find the color of a point on the picture of the object

Snipaste_2022-07-03_18-27-41.png

The traditional method needs to obtain the three-dimensional model of the object first , And build a three-dimensional scene , Then the color value of each region is obtained by calculation .

We can render by building a patch model , It can also be rendered by volume integration .

Snipaste_2022-07-03_18-28-02.png

Snipaste_2022-07-03_18-28-10.png

Here we consider two basic things , One is an object , The other is the observer ,

• The model of an object is physical , It does not change with the perspective of observation ;
• The observer observes the object model from various angles , It can be simplified into an angle .

and NeRF The way is to grasp these two key points , Abstract the rendering process of 3D objects into a function , The difference from traditional methods is , It no longer relies on explicit modeling of 3D objects .

Neural Radiance Fields & Multi-layer perception Neural Radiance Fields (NeRF) is a popular view synthesis technique that represents a scene as a continuous volumetric function, parameterized by multi-layer perception (MLP) that provide the volume density and view dependent emitted radiance at each location.

From the perspective of neural network , The rendering process is as follows

among , Represents model parameters , Represents the location of rendering , Represents the observation angle . The network training process is to shoot the same object from multiple perspectives , So as to optimize the parameters ,

therefore , It can be considered that the surface information of the object is encoded in the network parameters . The rendering of the new angle can be calculated through this set of parameters .

NeRF The problem of

They often fail to accurately capture and reproduce the appearance of glossy surfaces

Snipaste_2022-07-03_18-28-17.png

namely NeRF Poor treatment of smooth surfaces . The reason is , Because NeRF The processing of surface texture depends on interpolation

Snipaste_2022-07-03_18-28-25.png

Why do you say that? ？ Because the original function is a continuous function

therefore , Without special mechanism , When When it comes to gradients , Output It must also be gradual . Macroscopically, it will show “ interpolation ” The effect of . Reflected in the rendering results , There is no clear boundary for the color of the texture , It looks very vague .

The new method

The main idea

The new method learns the three-dimensional structure of the object surface , No longer rely on interpolation algorithm when rendering color

Snipaste_2022-07-03_18-28-35.png

The core of the three-dimensional structure here is the direction of the local normal .

How important are normals ？

Give an example to illustrate this problem , If you render an object in the traditional way , In addition to its 3D surface structure and texture map color , You also need its surface normal

Snipaste_2022-07-03_18-28-41.png

If you change its normal direction , You will get a completely different rendering effect

Snipaste_2022-07-03_18-28-51.png

The reason for this phenomenon can be explained by the following figure , Considering the ambient light , The color of a point of an object can be simplified into two factors

among , Represents the diffusion of materials （diffuse） Color , This color depends on the nature of the object itself , Represents the color brought by ambient light .

Snipaste_2022-07-03_18-28-25.png

Because the ambient light is reflected by objects , Therefore, it is inevitably affected by the joint action of illumination angle and surface normal .

Ref-NeRF The algorithm is in the process of learning model parameters , Learn the coupling structure of these normals and incident light at the same time by adding constraints .

Network structure

Snipaste_2022-07-03_18-28-59.png

In primitive NeRF In the method ,

• Input For location ;
• Input Represents the observation angle ;
• Output Represents material density ;
• Output Represents the color value ;
• Intermediate variable Represents bottleneck vector , It's kind of like ResNet Layer hopping transmission .

and Ref-NeRF The intermediate variables added by the method are ,

• ： The color produced by light and its own color ;
• ： Weighting of light color ;
• ： Local surface roughness ;
• ： local normal .

Why are so many things introduced besides normals ？ It's easy to understand .

• ： Control the value and proportion of external light and its own color ;
• ： After considering the surface normal , Local color value “ great ” The ground is affected by the normal , But in practice , Because the surface of the object is not absolutely smooth , This leads to a great difference between the actual results and the theoretical results . In this network , The rougher the surface , Then the smoother it is , Smoothing is done by fitting vMF Distributed implementation .

We introduce a technique,which we call an Integrated Directional Encoding (IDE), that enables the directional MLP to efficiently represent the function of outgoing radiance for materials with any continuously-valued roughness

Snipaste_2022-07-03_18-29-06.png

result

And NeRF Comparison of methods

Snipaste_2022-07-03_18-29-17.png

Snipaste_2022-07-03_18-29-24.png

so ,Ref The method can accurately estimate the surface normal structure of spherical and cylindrical structures , And what information is caused by ambient light . This endows the model with knowledge and learning “ Specular reflection ” The ability of .

Scene editing

Last , Because the model not only learns the surface information of the object , I also learned the information of ambient light , So we can change these two factors , To analyze 3D objects and scenes “ edit ”.

Snipaste_2022-07-03_18-29-30.png

We can edit the diffuse color of the car without affecting the specular reflections of its glossy paint

Snipaste_2022-07-03_18-29-38.png

We can plausibly modify the roughness of the car and material balls by manipulating the κ values used in the IDE