Render minecraft scenes into real scenes using NVIDIA GPU

GANcraft- Turn gamers into 3D The artist

NVIDIA And scientists at Cornell University introduced a hybrid unsupervised neural rendering process , It can effectively represent large and complex scenes in the world of three-dimensional elements . essentially ,3D Artists only need to build the simplest part , The algorithm will do the rest to build a realistic world . Researchers applied this hybrid neural rendering pipeline to Minecraft A piece of the world , To produce a more realistic Minecraft Scene version .

NVIDIA And the wider research community （pix2pix、pix2pixHD、MUNIT、SPADE） Previous work solved image to image conversion (im2im) The problem of —— Convert an image from one domain to another . At first glance , These methods seem to provide a simple solution for the task of transforming one world into another —— Convert one image at a time . However ,im2im Method does not maintain viewpoint consistency , Because they don't understand 3D The geometric , And each 2D Frames are generated independently . As can be seen from the following image , The results of these methods will produce jitter and sudden color and texture changes .

MUNIT	SPADE	wc-vid2vid	NSVF-W	GANcraft

Get into GANcraft, A direct pair 3D A new way to enter the world to operate .

Researchers at Research Explained in ：“ Because the photorealistic rendering of the block world created by the user does not exist at all , We have to pass
Indirect supervision to train the model .”

The working principle of this method is to randomly sample the camera view in the input block world , Then imagine what a realistic version of the view would look like . This is SPADE With the help of ,SPADE yes NVIDIA Previous work on image to image conversion , And it's popular GauGAN Key components in the demonstration . GANcraft By using a style adjustment network that can eliminate the ambiguity between world structure and rendering style , Overcome these generated “ False reality ” View inconsistency of . This makes GANcraft It can generate output video with consistent view , And different styles as shown in the figure ！

Although the research results are in Minecraft Has been shown , But this method is applicable to other 3D A piece of the world , For example, stereo pixels . The potential to shorten the time and expertise required to build a high-definition world increases the value of this research . It can help game developers 、CGI Artists and the animation industry reduce the time it takes to build these large and impressive worlds .

GANcraft stay Imaginaire In the library . The library is optimized for the training of generation model and generation countermeasure network , Support more GPU、 Multi node and automatic hybrid accuracy training . NVIDIA To make the 10 The implementation of many different research work and the pre training model have been released . as time goes on , This library will continue to update the updated works .