狗仔队:表面编辑多视点图像处理

摘要

Image processing pipeline with a wide variety of sophisticated filters and effects.Convert a single effect to an operation on 3D surface geometry,Will inevitably produce a custom algorithm.相反,We proposed a generic back-end optimization,Allows the user to edit an input 3D surface by simply selecting an off-the-shelf image processing filter.We do this by building a differentiable triangle mesh renderer,Through the renderer,We can backpropagate changes in the image domain to the 3D mesh vertex positions.A given image processing technique is applied to the entire shape via random snapshots of the shape：因此,我们称我们的方法为Paparazzi.We provide a simple and important design considerations,以构建PaparazziThe renderer and optimization algorithm.The power of this render-based surface editing is demonstrated by the various image processing filters we apply.Each application uses an implementation of off-the-shelf image processing methods,Without the need for the core ofPaparazzi算法进行修改.

CCS的概念.- 计算方法 → 网格模型.

其他关键词和短语：几何处理、Surface editing、基于图像的建模、几何变形、逆向图形

1 INTRODUCTION

Decades of research of digital image processing,Finally produced a large number of complex filters and effects.These filters are not only pre- and post-processing of other techniques in the image processing pipeline,A useful tool is also a graphic designer,For consumers and social media users provide satisfactory results.Many such filters rely heavily on the regular structure of the pixel grid.例如,Convolutional neural networks exploit this regularity to implement high-level advanced filtering operations,如神经风格转移.

Although some simple image processing filter（Such as Laplacian smoothing）Can be directly used in 3 d geometry processing,But building more complex filter analogue often need to deal with the special circumstances,To adapt to the topology structure of 3 d surface data、Bending measurement and irregular triangular grid combination.此外,Many image processing methods are difficult to redefine for 3D geometry.例如,The artistic style of painting can be effectively captured and transferred between images,But how to transfer the style of two-dimensional painting to three-dimensional surfaces is not very clear.

在本文中,We have developed a new kind of mechanical,即Paparazzi,to simultaneously extend a large number of image editing techniques to 3D shapes.The key idea is to modify the input 3D surface mesh by applying the required image processing techniques to many rendered snapshots of the shape（因此称为Paparazzi）.PaparazziIs the core of our rendering process differential,It allows to propagate changes in the image field to changes in mesh vertex positions.We first construct a stochastic multi-view optimization algorithm,Used for image processing based on energy minimization technique.然后,We further promote this algorithm,To adapt to the general iterative image processing filter.The renderer and their parameters are carefully constructed,To consider the view sample、Observation and vagueness of the shadow.Both our optimized intermediate meshes and the output triangular meshes are filtered,To ensure the water tightness,It is convenient to be used in downstream of the geometry processing,如3D打印（见图1中3D打印的L0Smooth frog illustrations）.我们展示了PaparazziThe versatility and the characteristics of plug and play,Will a few image filtering technology to3D形状,Including heuristic filter、量化、超级像素、L0Smooth style translation and nervous.有了Paparazzi,We simply by insertion of the existing,These images filtering technology popularization in geometry.

图1. PaparazziOn the 3 d shape plug-and-play image processing algorithm is realized.例如,The shape of the pixel to produce a Mosaic style;L0Specification make shape sheet surface,But retain the characteristics such as nose;Style transfer synthesize the artistic style of painting to the geometry.请注意,These pictures just to show2D效果,它们不在PaparazziThe optimization of the loop.

2 RELATED WORK

The theme of our work involves visual calculation,包括渲染、Computer vision and geometry processing.We focus our discussion on methodological or applied similar approaches.

差异化渲染. Rendering is a given scene geometry、材料、The lighting and observation conditions of information,The forward process of image synthesis.Solving this inverse problem is tantamount to solving computer vision.Loper和Black[2014]Put forward a complete rendering engine of differential,OpenDR,使用自动微分.The renderer for any input parameters are differentiable–Is not only the geometric,Therefore more common than we.For many of them, though the grid vertex differential,Show than the speed of naive finite difference is quite big,But in the case of this paper consider,Our analytical derivatives are orders of magnitude faster than their methods（见图2）.Liu等人[2017]A neural network architecture is proposed to approximate the forward image formation process,And predict intrinsic parameters from a single image、形状、Light and material.The neural network method is separable,And use existing data to achieve reasonable material editing results.然而,It is an approximate,For each task and each rendering parameters,Need a lot of training work.Many other differentiable or reversible renderers have been built,Used to estimate the material/Micro geometry[Gkioulekas等人,2013;Zhao 2014]或照明条件[Marschner和Greenberg 1997;Ramamoorthi和Han- rahan 2001].Although our lighting conditions and the material is fairly mild（Three directions of smooth shadow）,But we differentiate the whole image at all mesh vertex positions.
图2. We compare our method with two kinds of method based on automatic identification of each iteration of the running time（256×256图像）.Our approach is faster,扩展性更好.

Our analytical derivative is faster than existing automatic differentiation frameworks、规模更大.OpenDR（正向模式）[Loper和Black 2014]和TensorFlow 3D Mesh Renderer（反向模式,Also called back propagation）[Genova等人,2018].在一台机器上,Paparazzi可以处理超过100,000个变量的问题,但是OpenDR和TensorFlowRunning out of memory on problems with thousands and hundreds of variables respectively.在图2中,Our operation time（在256×256的图像上）要快几个数量级.

Based on the surface of the image editing. In geometric modeling,Many previous methods proposed interactive or automatic methods,By directly specify the rendering of the shapes to edit a[Kerautret等人,2005;Tosun等人,2007;Van Overveld,1996].例如,Gingold和Zorin[2008]Allows the user to draw darkened and lightened strokes on a surface rendered with a single light source.In order to overcome the shadow of ambiguity–This is a tricky problem common to all methods of getting shape from shadows–They chose to the existing surface change the minimum deformation.相反,We overcome this ambiguity by increasing the complexity of the lighting.Schüller等人[2014]利用Lambertian表面的bas-reliefFuzziness to create bounded the thickness of the surface,The surface from a particular perspective with a given surface with the same appearance.A unique contribution of our work is,We use stochastic gradient descent to optimize the desired appearance on all views of a surface.

Image-based methods are widely used for mesh simplification and accelerated rendering[Weier等人,2017].These methods re-simplify polygon meshes to improve rendering efficiency,But retain their perceptual appearance[El-Sana等人,1999;Hoppe,1997;Lindstrom和Turk,2000;Luebke和Erikson,1997;Luebke和Hallen,2001;Williams等人,2003;Xia和Varshney,1996] .The success of image-driven simplification demonstrates the power of image-driven approaches,But just as a measure of the.Our method is further,Using the render similarity,Will be a lot of image processing technology to surface editing.

Kato等人[2017]Through the application on the rendering image style transfer,Transferring nerve style are generalized to three-dimensional grid.在高层次上,他们的方法与Paparazzi相似,Because they will spread to the geometry image gradient,But their derivative is an approximate,And our derivative is the analysis of the.特别是,What they consider is whether a pixel is covered by a certain triangle,This requires a non-differentiated step function that approximates the motion of mesh vertices in 3D.相反,We consider the direction of the triangle（Each facet normals）Under the infinitesimal perturbation of grid vertex how to change.It can capture the continuous variation of the each pixel color,And implement the analytical derivative.Kato等人[2017]does not prevent the inevitable self-handover during large deformations（See the red）.Since the handover may result in differences or The secondary optimization results.这些差异使Paparazzi成为一个 这些差异使得PaparazziBecome a more general image driven method,Used to create high quality,Even can make three-dimensional objects.Can be made in 3 d object.

The shadow of shape. 从拍摄的（Or apply colours to a drawing）To recover the image geometric shape is called "Look from the shadow shape "问题.This child field itself is very broad,So we follow the existing research[Prados和Faugeras 2006;Zhang等人1999],Focus on the most relevant methods.In the case of insufficient data or unreliable,Shape algorithms from shadows often fall back on the assumption of regularization,Such as surface smoothness[Barron和Malik 2015],And therefore produce less detailed model.In the presence of every pixel depth information,Single view shadow shape problem easier,在这种情况下,Reverse rendering geometry can be used to refine depth,To match the shadow image[Or-El等人,2016;Wu等人,2014] .If given multiple depth and shadow images from different views,Based on the depth of the shadow refinement can be extended to the whole shape reconstruction refinement[Choe等人,2017;Robertini等人,2017;Wu等人,2011] .Gargallo等人[2007]Accurately distinguish reprojection error functions with respect to unknown surfaces.Delaunoy和Prados[2011]To expand its to minimize based on image regularization item,In order to help more view surface reconstruction.All these methods of getting shape from shadows are based on the assumption,i.e. the input data is captured by a fully realized physical shape–No matter how unreliable.Our problem is similar to geometric refinement based on multi-view shading,但有一个主要的区别–We can get a general underlying geometric said.We use this channel to develop a more powerful framework,To promote the various image processing directly to the 3 d,Not just the geometric refining.

Single use geometry processing filter. In our results,We will demonstrated by simplyPaparazziAttached to the existing image processing code（例如Skimage [Van der Walt et al. 2014]）,Examples of various filter was applied to the geometric.我们的结果表明,We successfully transfer the effect：例如,[Xu等人,2011]Create the flake constant images,通过Paparazzi,We use their method to create flaky constant looking geometry（见图3Single view example）.We are used to demonstrate some of the image processing filter
Some of the image processing filters we used for demonstration purposes have previously been converted to single-purpose filters for triangular meshes.例如,He和Schaefer[2013]The introduction of a novel based on the edge of the Laplace,将L0Regularization applied to the grid.同样,To create a 3D-like guided filter for the mesh[He等人,2010],Zhang等人[2015]A triangular clustering method tailored specifically for guided filters is designed.Extending texture synthesis to 3D geometry has been an active area of ​​research[Dumas等人,2015;Gu等人,2002;Knöppel等人,2015.Lai等人,2005;Landreneau和Schaefer,2010;Turk,1991;Wei和Levoy,2001],Typical challenges lie in the consideration of curvature and irregular mesh discretization.

图 3. Bunny 进行了优化,Therefore the rendering image gradient in a single view L0 Sense is the smallest.

Our technical contribution is not for any specific mesh filter or application（如去噪）Increase technical level,But rather to provide an algorithm,In order to provide a general、Plug and play machine,Directly to a large number of filters applied to image processing3D.我们根据PaparazziThe extent to which image processing effects are correctly applied on the input geometry to evaluate our results.

3 OVERVIEW

PaparazziIs a generic machine,It allows users to apply image processing filters to 3D geometry,Without the need for surface filtering again,Don't even need to implement new triangle mesh code.The input to our method is a non-self-intersecting Manny triangular mesh and a specified image processing technique.The output is the grid of selfing deformation,Its appearance through the specific processing.其核心观点是,If we can pull the gradient from the rendered image back to the vertex,Then we can perform gradient-based optimization on vertex positions.We first describes a very good program,where the specified image processing techniques are described as energy optimization in the image domain.随后,我们表明,Based on energy method for us make a little change,We can generalize to the category of iterative image processing techniques.

3.1 Image filter based on energy

Many image editing algorithm can be expressed as differential、Image in the field of energyE的最小化.在理想情况下,We will minimize any such energy spread to the surface,Consider the integral of this energy for all possible cameras "视图 "Space rendering image.

其中 Ri Is to get the vertex V The grid mapping to the image function.

使用链式法则,Minimization can be directly written about vertex gradient descent：

View space can be adjusted to meet the needs of specific problems.例如,It can be small to a positive camera,Or to all of the space camera,There are a certain number of geometry is visible.We will discuss for a good default choice until the first6节.

考虑一下图4Toy examples of energy-based image editing algorithms seen in,The energyEOnly with another shape renderingL2距离.在优化过程中,We only consider a sphere of a single view.After gradient descent,The geometry of a sphere is deformation,thus making this one view imperceptibly similar to the source image.For a single view,We only change affect the view rendering method of vertex,This makes the result looks like decals.

图4. A sphere is deformed to match the source image from the cameraI ′.

图5. In the multiple view,We each iteration with a view to sampling.

Jacobian ∂Ri/∂VThe existence of the exposed the rendererR的主要要求：Relative to the vertex positions of differential can be.在本文中,We put forward a renderer as simple as possible,It is under mild assumptions analysis,And can effectively generate high quality geometry（见第5节）.

3.2 To view more stochastic optimization

When we see a single view,解析导数 ∂R/∂V 可以通过方程 (1) To promote the image processing algorithm directly to the geometric shape,But evaluating such integrals on continuous space or view distribution is challenging.

We approach this problem by borrowing tools from the machine learning community,They have extensively applied gradient descent to energies involving integrals or large sums when training deep networks[Bottou等人,2016;Ruder 2016].We did not attempt to precise calculation of composite gradient,But using the stochastic gradient descent（SGD）,With a small number of gradient to update the view geometry,Is one less.As common in machine learning literature,We apply momentum to normalize the noise introduced using stochastic gradients,并使用Nesterov-Adam方法[Dozat 2016]To improve the general performance,This method is variation gradient descent,Is a combination of momentum andNes-terov的加速梯度.Our random multiview optimization algorithm1中进行了总结.

As the grid according to the optimized deformation,The quality of the triangle may reduce,并且可能（And inevitably）Occurred since the intersection.我们在 4.3 Section discusses the importance of the staggered mesh quality improvement stage in the optimization loop.

3.3 Iterative image filter

只需稍作修改,We can then generalize our method from energy-based deformation to the domain of iterative filters,Generally defined as an iterative process in the field of image.

We will through derivative ∂E/∂R Replace with poor ΔR :=f − R to replace the energy gradient with an update caused by a single iteration of the filter.The grid vertex positions update into：

When a single application of iterative filtering does not modify the image too much,This summary will work.在我们的实验中,This is close enough to the smoothness,allows our method to converge to a geometry that matches the filter results in different views1.If a single application filter has a dramatic effect,So we can through the use of the optimization of the small stepγ来适应.因此,Generated by the algorithm is 2.

Before the show results（第7节）,We'll cover the considerations we made when designing our renderers and parameters.

4 设计注意事项

By processing the geometry rendering,Image processing techniques can be applied to rendered images of pixels in their native form.这使得 Paparazzi Can immediately to the categories of image processing technology,But will shift the burden to design on the render Settings,The setup faithfully captures geometry and presents it to image processing techniques in a meaningful way.Where and how we render geometry will have a big impact on the quality of the result.

4.1 相机采样

A good camera placement strategy should be in the same probability "看到 "On the surface of each part.A surface patch that has never been seen by any camera will not be altered.另一方面,A surface patch seen by too many cameras will update faster than other surface areas,And cause discontinuity between surface patches.

Based on these two standards：Full coverage and uniform sampling,Paparazzi在距离σThe offset surface evenly to the camera samples,The point with a given shape has a fixed distance,Along the vertical normals facing inward.This arrangement ensures that our prejudice against certain views less,In the sharp edges are smooth camera view,And for most shapes have comprehensive coverage.Because we are using the orthogonal camera,Therefore increase and decreaseσOnly affects nearly flat.我们将σSet to shape boundary box of the diameter of the small deformation5%,Large deformation for25%（See the value of each experiment the first6节）.

4.2 光照和阴影

Our image-driven surface editor is designed to input and output 3D shapes,So the intermediate rendering quality only matters if we achieve the desired output geometry.我们为PaparazziPut forward a default renderer as simple as possible,But pay special attention to avoid ambiguity of light and shadow,So as not to cause artifacts in the process of optimization.

The shadow of fuzziness. 众所周知,Light in a single direction is not enough to distinguish convex and concave shapes and slope directions（见,例如,[Liu and Todd 2004]）.Just as the ambiguity of this shadow can confuse human observers,It can also confusePaparazzi的优化.一个原因是,The single direction of the light is not enough to distinguish between the convex/Concave shape and slope direction（见图6）.

图6. The shadow of fuzziness.凸/Concave shapes may produce the same image under a single direction of light（中间）.Adding complexity to lighting can solve the problem of blurry shadows.

Our simple solution by the photometric stereo [Woodham 1980] 的启发,Is to increase the complexity of the lighting.Through the three specified respectively with R、G、B Shaft alignment direction of color lights,We effectively apply colours to a drawing surface normal vector of the image.This avoids the shadow blur.

Gouraud模糊性. If we follow the smooth and interpolated per-vertex lighting within a triangle common in computer graphics/Constant practice,There will be a more subtle,But a crucial fuzziness.When rendering a triangular grid（Especially in the case of low resolution）,Gouraud着色[1971]或Phong着色[1975]Makes the shape appear smoother than the actual flaky linear geometry.Although this delusion are very convenient for efficient rendering,But its inherent average will lead to a fuzzy.A rough surface geometry inGouraudStill can produce a smooth shadow rendering.我们将此称为Gouraud模糊性.在PaparazziUsed in the optimization process ofGouraudThe shadow will immediately produce a empty space,Lead to numerical problems and bad concave and convex geometry2（见图7）.相反,We recommend the use of planar shadow.从某种意义上说,This is the most realistic rendering of a triangular mesh piecewise linear geometry.

图7. Gouraud模糊性.Given an uneven sphere（左）,我们在GouraudMinimize shadowim- age Dirichlet能量,To get a smooth sphere（中）.More smooth region render,我们观察到Gouraud模糊性,i.e. the rendering of a non-smooth sphere is very similar to the rendering of a smooth sphere（左栏）,But smooth shadow shows the difference（右栏）.

4.3 网格质量

到目前为止,We render select enough forPaparazziOf geometry on the surface of the small change,But with the continuous deformation of the grid,The quality of a single triangle will decline,Even degradation.此外,May also occur between local and global cross.For us to consider the many image processing filter,We want sharp creases and corners during surface deformation,This without a grid, it may not be able to achieve.

These challenges and specifications are similar to the remeshing required during surface tracking fluid simulations.We borrowed one from the community the most advanced tools El Topo [Brochu and Bridson 2009],This is a package for robust explicit surface tracking using triangular meshes.它使 Paparazzi Ability to generate multiple waterproof meshes without self-intersection,And in the region of the high curvature refined grid,This allows us to introduce sharp features without worrying about mesh locking.在图 8 中,We can see the shape optimization need El Topo To reduce the poor quality of the grid and the intersection problem,Even if the rendering result comparable.

图8. 没有El TopoShape optimization of（底部）May lead to self cross（红色）,尽管事实上,The renderings with useEl Topo的相似（顶部）.此外,使用El TopoThe error of the optimization result is low（左边）.请注意,The peak figure is weEl Topo的地方.

El TopoTo deal with two kinds ofPaparazziIs very important for the success of operation：To maintain manifold、The key to the intersection of grid operation and the triangle quality related operations.我们向El TopoProvided in the current grid vertex and,And a new expectations vertex positions.El TopoCheck if the triangles will collide or get too close in a continuous motion from the current position to the desired position.This may cause repulsion or topology change,Depending on the user defined threshold.In order to improve the quality of the grid,So as to improve the robustness of collision detection,El TopoThe standard grid improvement actions,Such as edge segmentation and edge flip,Increase the aspect ratio of triangles without affecting the overall topology of the mesh.El TopoAlso carried out on the grid subdivision and differentiation,By moving the edge Angle in a user defined interval,Respectively, improve the quality of high curvature and low curvature area grid.

Reshape and collision handling are critical to maintaining high quality meshes during and after optimization,But it is also very time consuming–Especially compared to our derivative calculation.这在图9中可见,我们可以看到El TopoIn the total run time dominant.Because the deformation between any individual iterations is generally small,In practice, each of us30Iteration is called aEl Topo,Provides an empirically determined balance between computation time and optimized performance.

图9. We showed our total run time break down,Do not include image processing part.The upper part is derivative calculation of time;下半部分是El Topo运行时间的1/30.

5 可区分的渲染器

到目前为止,我们已经讨论了Paparazzi的设计考虑.Due to the intermediate rendering is not output,We can be flexible to design a suitable renderer,解决上述挑战,更重要的是,它是可微的.Especially we proposed a renderer of differential,Enable us to analysis and calculation∂R/∂V,And image processing are generalized to three-dimensional geometry.

5.1 可见性

Away from the silhouette and shade contour,Rendering of triangular meshes with flat shadows is continuous.它几乎在任何地方都是可微的：In the image plane located at all points within the triangle,But not including triangle edges or ends（A collection of the measurement is zero）.因此,We assume that small changes in surface points do not change visibility,Because in practice we only have limited image resolution on the computer.

In our optimization loop eventually produce great vertex disturbance,Visibility may change.幸运的是,Due to the real-time rendering engineOpenGL中的Z-buffering的效率,Updating visibility can be handled efficiently by re-rendering the shape each iteration.

5.2 解析导数

In view of our design on the surface of local lighting and lambert selection,我们用相应的 RGB 颜色 {cRi ,cGi ,cBi } ∈ [0,1] Rendering by direction ^ℓi （R3 中的单位向量）指向的 m A directional light,The output color pixel p 处的 {r Rp ,r Gp ,r Bp } ∈ [0,1] 由下式计算

其中,ˆnj是R3中的一个单位向量,On behalf of the triangular gridV的第jThe normal,第jA plane is a pixelpThe recent face.

在不失一般性的情况下,We only write the red parts in the derivationr Rp,因为r Gp ,r BpHave the same expression.We can analytically differentiate this formula based on vertex positions,Each line form the sparse matrix of Jacob∂r Rp /∂V∈R3 |V |.Only each vertexvk∈R3在第jThe position of the Angle of the triangle to contribute.

最后,三角形顶点vk ,nj^hjk vkOn the face normals3×3雅各布系数ˆnj,Can use the method to calculate.请注意,Move in the plane of the trianglevk不会改变ˆnj.另外,在极限情况下,沿ˆnjMobile only changeˆnj在hjk方向的位置,其中hjk∈R3是 "高度 "向量：From the edge to the Angle ofvkThe shortest vector.这意味着Jacobian必须是hjk ˆnTjA scalar multiples.The change and thevkOn the edge of the distance∥hjk ∥成反比,这意味着.

6 实施

在我们的实验中,We normalize the shape to fit a unit radius cube centered at the origin,And the shape of the sample,以达到105-106个顶点,To capture the geometric details.默认情况下,We use a square、具有0.5Wide-field camera,放置在σ=0.1的偏移处,The unit isOpenGLThe standard view volume.The surface of the offset mesh have103-104个顶点.默认情况下,We use red along each axis of the local camera、绿、Directional light blue three colors,This is equivalent to the camera frame render the surface normals.

我们在PythonThe use of vectorNumpyOperation and callOpenGLTo render and rasterize to implement our derivative calculation.我们使用libigl [Jacobson et al. 2016]和Mesh- mixer [Schmidt and Singh 2010]For grid sampling and offset surface calculation.We are equipped with Intel xeon3.5GHz CPU、64GB内存和NVIDIA GeForce GTX 1080 GPU的LinuxThe realization of the workstation to test our.

6.1 The ready-made image processing filter

我们将PaparazziDesigned with the existing image processing filter plug and play.We can with the minimum effort using open、The realization of the ready-made image spatial filter.为了评估我们的方法,We use some ready-made image filter.We use the quick guide filterPython实现[He and Sun 2015],可以在github.com/swehrwein/python-guided-filter找到.对于SLIC超级像素[Achanta等人,2012],我们使用流行的Python图像处理库Skimage中的实现.我们将github.com/soundsilence/ImageSmoothing中的Matlab实现的L0-smoothingThe image smooth[Xu等人,2011]翻译成Python.For nerve style transformation[Gatys等人2016],We follow the correspondingPyTorch[Paszke等人2017]教程,And adapt to extract∂E/∂Ri梯度.Our own implementation of simple image quantization with a fixed palette（见[Ozturk等人,2014]中的评论）.

Will these filters used in three-dimensional geometry,不需要修改Paparazzi算法.The caller or provide∂E/∂RiTo use gradient algorithm1The method based on energy,Either provide filterfAs a function handle to use algorithm2的迭代方法.从用户的角度来看,Try all kinds of filters is easy.在图10中,我们展示了PaparazziHow to produce different results for the various smoothing filters we tested.Each result respects the intent of a specific image processing filter,But now used in a 3 d surface.

图10. PaparazziAllows direct generalization to the 3 d image processing,So different image editing effects can be directly converted into 3D shapes.

7 Assess and discuss

在表1中,We break down our running time in the form of subroutines：导数计算、Image processing and the use ofEl TopoClean up the grid.Our derivative computations are orders of magnitude faster than previous methods（见图2）.Grid clean is the bottleneck of high resolution grid（见图9）.As a result of our multiple view operation processing local patches of rendering,The runtime performance of a specific input image processing method is amplified by our method（例如,A simple quantitative far more quickly than the nerve transfer style）.

For filter based on energy,Evaluating the synthetic multi-view energy requires rendering and evaluation from all possible camera views.i.e. approximate computation in each iteration with a limited number of views,Is too expensive.相反,In order to assess the convergence behavior,We can set a camera on a fixed view,And in multiview optimization random decrease（未测量的）Comprehensive energy when evaluate the visible.Specific rendering view does not mean more energy energy value,But the convergence behavior means multiview convergence of energy.在插图中,We show that the convergence of neural style transfer energy situation.

7.1 The evaluation of image filter

我们根据PaparazziThe ability to reproduce the effect of a 2D filter on a 3D shape to assess it,Not according to it in any specific application（例如去噪）The success of the specific areas of the.在图10中我们看到,Changing the image processing filter does change the resulting edit shape.Guided filters correctly implement smoothing that preserves edges;To quantify the surface patches with predefined normal3对齐;Super pixels to create a super surface;L0-Smooth is the result of the flake of plane geometry.我们可以看到,These filters are correctly transferred into the 3D geometry in a plug-and-play fashion.同时,Our reshaping technology ensures that the output mesh is impeccable.

We first consider a simple and powerful differentiable energy——L2 像素差.Because of its derivative ∂E/∂R 是已知的,So we apply the algorithm 1 Will this energy to 3D 形状.By caching a geometry rendering,We can use this energy minimization to transfer its appearance to another geometry.With a dedicated grid transmission tool（例如,[Takayama et al. 2011]）相比,We don't need to source geometry is another triangle grid：Only we can render anything.在图 11 中,我们可以通过最小化 L2 图像差异∥R (P )-R (V )∥2 The details from point cloud P Transferred to the triangular grid V.我们使用简单的 splat 渲染,But this example will work right away for more advanced point cloud rendering（例如,参见 [Kobbelt and Botsch 2004]）.

图11. We through the renderingR (P )点云,The geometric details from the input point cloudPTransferred to the input shapeV.

The source geometry can be a grid with defect,Such as selfing and hole.在图12中,We transfer the appearance of the triangle soup to use robust signed distance offset[Barill等人,2018]Create a smooth surface reconstruction of above.The result is a new grid not leak,With the appearance of the chaos of the input,Mesh repair methods are difficult to keep the appearance before[Attene 2010, 2016].The grid for now3D打印.

图12. We can create a rough proxy,Then transfer to repair a broken application details section and the intersection of the grid.These defects inMeshmMixer[Schmidt和Singh 2010]Is the visual,And prove the special grid cleaning method is difficult.

在下面的图13-17中,On the left side of the image is used as a reference to display the corresponding image processing,Is not used to make surface edit the results.根据结构,Our 3 d input and output reflects theHertzmann等人[2001]的 “类比”,But unlike the method,We can direct access to the underlying image processing algorithm.

We are now exploring a more complex energy–Neural style energy.最近,受卷积神经网络（CNN）The power of inspiration[Krizhevsky等人,2012],Nerve transfer style has become a popular tool,Used to transfer the artistic style from painting to other image[Gatys等人,2016].The goal is to generate a stylized image given a content image and a reference style image.Gatys等人[2016]Total energy is defined as the sum of the energy content and style,The output image where the content energy encourages stylization has an image structure similar to the content image,Style energy encourages output images to have similar characteristics to reference style images.请注意,These features is to useCNNDifferent layers of filter response to define.

Artistic style will be transferred to the three-dimensional geometry is a challenge,Because the 2 d drawing style in 3 d to redefine it is not clear.有了Paparazzi,We can generalize it by applying image neural style transfer on the rendered graph.Because image gradient can distinguishCNN来实现,We can use the algorithm1To generate a stylized shape.在图13中,PaparazziThrough the growth of the geometric two-dimensional drawing style transferred to 3 d（我们在附录AProvide details about the implementation of the image style of nerve）.

图13. We through the manipulation of the vertex positions,Minimize local rendering style energy,To convert nerve style extended to 3 d.

A large proportion of image processing algorithms are not based on energy minimization,But the application of iterative procedure.These algorithms may not have a clear energy,或者即使有,Or may not have an easy to calculate the gradient of.幸运的是,PaparazziProvides an effortless way to generalize using algorithms2All kinds of iterative image filter.High-level thinking is to be the time when rendering an image updates,and update the shape once based on the image changes caused by the image update.

导向滤波器[He等人,2010]Calculate the filtered image output by considering the content of the guide image,Guided imagery can be another image or input itself.He等人[2010]表明,Guide the filter is very effective in various image processing applications,Including the edge smooth perception、细节增强、Image feather, etc.在图14中,我们用[He and Sun 2015]Accelerated method is proposed to apply in edge awareness smoothing filter.We set up the boot image for the input,滤波器参数为r =4,ε =0.02.Through the filter insertion algorithm2中的滤波函数f,我们可以看到,Guided filters smooth out 3D shapes and preserve sharp features.

图14. We will quickly lead filter extended to 3 d,Edge and implements the protection effect of smooth.

In addition to retain the edge of the smooth,We are interested in using image filters to create different visual effects on geometry.A simple style option is image quantitative,This is a kind of image compression technology,Will a series of color value compressed into a single,Only a small part of the color image[Ozturk等人,2014].同样,By changing the algorithm2中的过滤器f,We can use a predefined set of color4对3DQuantify the shape（见图15）.请注意,These color is encoded in world coordinates,Therefore shape quantification is dependent on the direction of,Need to apply colours to a drawing normals in world coordinates,This is different from other filters that render normals in local coordinates.

图15. Images are applied to quantify the geometry,And the surface patches toward the predefined palette

Another kind of pixel method,But at the same time based on color and spatial information,Is a super pixel.在图16中,We use a simple linear iterative clustering（SLIC）[Achanta等人,2012],It adapt to thek-meansTo split pixel,In shape to create “Super face”.

图16. SLICSuper pixel method is applied to the three-dimensional objects,Results in small surface patches appeared on the shape

最后但并非最不重要的是,We consider is to make the image gradientL0Guidelines to minimize the filter[Xu et al. 2011].几十年来,L0Criterion has been a popular image and signal processing tool,Because it is the direct measure of signal sparse degree.然而,由于L0Criterion of discreteness and combination,It may be difficult to optimize.Xu等人[2011]An iterative image optimization method is proposed,以最小化L0梯度,And keep the edge of the、Flake constant filtering effect.通过算法2,We can simply using the iterative procedure,将L0Criterion effect are generalized to three-dimensional shape,And make it become a flake plane,This is the image of three-dimensional analogue of flake constant.In the image three dimensional simulation（见图17）.

图 17. We minimize the image gradient L0 范数,And to encourage output shape（蓝色）Is piecewise plane.

8 限制和未来工作

PaparazziSample a precomputed offset surface for the camera's position.This means that heavily occluded or tight surface lumens will not receive edits（例如,In the crocodile's mouth）.This also means that the shape is implicitly trapped in its original offset surface cage.Removing the constraints of this cage and predicting changes in visibility will help create large shape deformations.For a more rigorous and more sure Image energy,It is important to aim at the direction of the camera,In order to encourage the overlapping area of each view consistency.同时,We have only presented analytical derivatives for flat shaded triangular meshes;Similar derivative can be derived form other,Such as subdivision surface orNURBS模型.PaparazziThe differentiation of several orders of magnitude faster than the previous job.Several orders of magnitude faster before work.在未来的工作中,We hope to further improve 在未来的工作中,We hope to optimize parallelism and improvePaparazziTo further improve the performance of its performance.We hope to improve this further by exploiting the parallelism of random multi-view optimization and improving collision detection required for dynamic meshingPaparazzi的性能. Dynamic meshing for collision detection（目前,El Topo–Used as a black box–主导着Paparazzi的性能.目前,El Topo–作为一个黑盒子–The operation of the dominate our time,见图9）.

在其核心部分,PaparazziIs a can fine-tune the renderer,With stochastic gradient bleaching process multiple points of view,Images can be change back propagation to the 3 d surface.PaparazziFor the three dimensional interpretation of the two-dimensional filter,But it can be a useful tool for studying other filters that have no direct 3D interpretation.将PaparazziExtended to operate with global illumination and textures as well as more complex lighting models,May beneficial to the application need realistic rendering,例如图像分类.In our style of nerve transfer instance,我们只展示了PaparazziA small sign of greater possibility in transferring the success of image-based deep learning to 3D surface geometry.PaparazziShows the rendering of the utility,Not only for visual,But also a way to edit 3 d shape.考虑PaparazziThe other ways to be able to affect and interact with the geometry processing pipeline are exciting.