Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis

Last update: Dec 27, 2022

Related tags

Overview

Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis

This repository contains the official implementation of DietNeRF, a system that reconstructs 3D scenes from a few posed photos.

Setup

We use the following folder structure:

dietnerf/
  logs/ (images, videos, checkpoints)
  data/
    nerf_synthetic/
  configs/ (run configuration files)
CLIP/ (Fork of OpenAI's clip repository with a wrapper)

Create conda environment:

conda create -n dietnerf python=3.9
conda activate dietnerf

Set up requirements and our fork of CLIP:

pip install -r requirements.txt
cd CLIP
pip install -e .

wandb login

Experiments on the Realistic Synthetic dataset

Realistic Synthetic experiments are implemented in the ./dietnerf subdirectory.

You need to download datasets from NeRF's Google Drive folder. The dataset was used in the original NeRF paper by Mildenhall et al. For example,

mkdir dietnerf/logs/ dietnerf/data/
cd dietnerf/data
pip install gdown
gdown --id 18JxhpWD-4ZmuFKLzKlAw-w5PpzZxXOcG -O nerf_synthetic.zip
unzip nerf_synthetic.zip
rm -r __MACOSX

Then, shrink images to 400x400:

python dietnerf/scripts/bulk_shrink_images.py "dietnerf/data/nerf_synthetic/*/*/*.png" dietnerf/data/nerf_synthetic_400_rgb/ True

These images are used for FID/KID computation. The dietnerf/run_nerf.py training and evaluation code automatically shrinks images with the --half_res argument.

Each experiment has a config file stored in dietnerf/configs/. Scripts in dietnerf/scripts/ can be run to train and evaluate models. Run these scripts from ./dietnerf. The scripts assume you are running one script at a time on a server with 8 NVIDIA GPUs.

cd dietnerf
export WANDB_ENTITY=
   
    

# NeRF baselines
sh scripts/run_synthetic_nerf_100v.sh
sh scripts/run_synthetic_nerf_8v.sh
sh scripts/run_synthetic_simplified_nerf_8v.sh

# DietNeRF with 8 observed views
sh scripts/run_synthetic_dietnerf_8v.sh
sh scripts/run_synthetic_dietnerf_ft_8v.sh

# NeRF and DietNeRF with partial observability
sh scripts/run_synthetic_unseen_side_14v.sh

Experiments on the DTU dataset

Coming soon. Our paper also fine-tunes pixelNeRF on DTU scenes for 1-shot view synthesis.

Citation and acknowledgements

If DietNeRF is relevant to your project, please cite our associated paper:

@InProceedings{Jain_2021_ICCV,
    author    = {Jain, Ajay and Tancik, Matthew and Abbeel, Pieter},
    title     = {Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis},
    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
    month     = {October},
    year      = {2021},
    pages     = {5885-5894}
}

This code is based on Yen-Chen Lin's PyTorch implementation of NeRF and the official pixelNeRF code.

Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis

Related tags

Overview

Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis

Setup

Experiments on the Realistic Synthetic dataset

Experiments on the DTU dataset

Citation and acknowledgements

Owner

Ajay Jain

Image inpainting using Gaussian Mixture Models

The datasets and code of ACL 2021 paper "Aspect-Category-Opinion-Sentiment Quadruple Extraction with Implicit Aspects and Opinions".

On Uncertainty, Tempering, and Data Augmentation in Bayesian Classification

This is a model made out of Neural Network specifically a Convolutional Neural Network model

Behind the Curtain: Learning Occluded Shapes for 3D Object Detection

Official PyTorch Implementation of Mask-aware IoU and maYOLACT Detector [BMVC2021]

Narya API allows you track soccer player from camera inputs, and evaluate them with an Expected Discounted Goal (EDG) Agent

Minimal PyTorch implementation of Generative Latent Optimization from the paper "Optimizing the Latent Space of Generative Networks"

Solve a Rubiks Cube using Python Opencv and Kociemba module

Train CPPNs as a Generative Model, using Generative Adversarial Networks and Variational Autoencoder techniques to produce high resolution images.

Codes for our IJCAI21 paper: Dialogue Discourse-Aware Graph Model and Data Augmentation for Meeting Summarization

darija <-> english dictionary

[CVPR 2021] MiVOS - Scribble to Mask module

QueryFuzz implements a metamorphic testing approach to test Datalog engines.

A Player for Kanye West's Stem Player. Sort of an emulator.

Weighted QMIX: Expanding Monotonic Value Function Factorisation

Official Pytorch implementation of "Learning to Estimate Robust 3D Human Mesh from In-the-Wild Crowded Scenes", CVPR 2022

Production First and Production Ready End-to-End Speech Recognition Toolkit

PyTorch implementation of paper "IBRNet: Learning Multi-View Image-Based Rendering", CVPR 2021.

[CVPR2021] The source code for our paper 《Removing the Background by Adding the Background: Towards Background Robust Self-supervised Video Representation Learning》.