Official Pytorch implementation for video neural representation (NeRV)

Last update: Dec 28, 2022

Related tags

Overview

NeRV: Neural Representations for Videos (NeurIPS 2021)

Project Page | Paper | UVG Data

Hao Chen, Bo He, Hanyu Wang, Yixuan Ren, Ser-Nam Lim, Abhinav Shrivastava
This is the official implementation of the paper "NeRV: Neural Representations for Videos ".

Get started

We run with Python 3.8, you can set up a conda environment with all dependencies like so:

pip install -r requirements.txt

High-Level structure

The code is organized as follows:

train_nerv.py includes a generic traiing routine.
model_nerv.py contains the dataloader and neural network architecure
data/ directory video/imae dataset, we provide big buck bunny here
checkpoint/ directory contains some pre-trained model on big buck bunny dataset
log files (tensorboard, txt, state_dict etc.) will be saved in output directory (specified by --outf)

Reproducing experiments

Training experiments

The NeRV-S experiment on 'big buck bunny' can be reproduced with

python train_nerv.py -e 300 --cycles 1  --lower-width 96 --num-blocks 1 --dataset bunny --frame_gap 1 \
    --outf bunny_ab --embed 1.25_40 --stem_dim_num 512_1  --reduction 2  --fc_hw_dim 9_16_26 --expansion 1  \
    --single_res --loss Fusion6   --warmup 0.2 --lr_type cosine  --strides 5 2 2 2 2  --conv_type conv \
    -b 1  --lr 0.0005 --norm none --act swish

Evaluation experiments

To evaluate pre-trained model, just add --eval_Only and specify model path with --weight, you can specify model quantization with --quant_bit [bit_lenght], yuo can test decoding speed with --eval_fps, below we preovide sample commends for NeRV-S on bunny dataset

python train_nerv.py -e 300 --cycles 1  --lower-width 96 --num-blocks 1 --dataset bunny --frame_gap 1 \
    --outf bunny_ab --embed 1.25_40 --stem_dim_num 512_1  --reduction 2  --fc_hw_dim 9_16_26 --expansion 1  \
    --single_res --loss Fusion6   --warmup 0.2 --lr_type cosine  --strides 5 2 2 2 2  --conv_type conv \
    -b 1  --lr 0.0005 --norm none  --act swish \
    --weight checkpoints/nerv_S.pth --eval_only

Dump predictions with pre-trained model

To evaluate pre-trained model, just add --eval_Only and specify model path with --weight

python train_nerv.py -e 300 --cycles 1  --lower-width 96 --num-blocks 1 --dataset bunny --frame_gap 1 \
    --outf bunny_ab --embed 1.25_40 --stem_dim_num 512_1  --reduction 2  --fc_hw_dim 9_16_26 --expansion 1  \
    --single_res --loss Fusion6   --warmup 0.2 --lr_type cosine  --strides 5 2 2 2 2  --conv_type conv \
    -b 1  --lr 0.0005 --norm none  --act swish \
   --weight checkpoints/nerv_S.pth --eval_only  --dump_images

Citation

If you find our work useful in your research, please cite:

@inproceedings{hao2021nerv,
    author = {Hao Chen, Bo He, Hanyu Wang, Yixuan Ren, Ser-Nam Lim, Abhinav Shrivastava },
    title = {NeRV: Neural Representations for Videos s},
    booktitle = {NeurIPS},
    year={2021}
}

Contact

If you have any questions, please feel free to email the authors.

Official Pytorch implementation for video neural representation (NeRV)

Related tags

Overview

NeRV: Neural Representations for Videos (NeurIPS 2021)

Project Page | Paper | UVG Data

Get started

High-Level structure

Reproducing experiments

Training experiments

Evaluation experiments

Dump predictions with pre-trained model

Citation

Contact

Owner

hao

POPPY (Physical Optics Propagation in Python) is a Python package that simulates physical optical propagation including diffraction

Unofficial implementation of Perceiver IO: A General Architecture for Structured Inputs & Outputs

Hepsiburada - Hepsiburada Urun Bilgisi Cekme

Model Zoo for MindSpore

FairMOT - A simple baseline for one-shot multi-object tracking

Generalized Data Weighting via Class-level Gradient Manipulation

(CVPR2021) DANNet: A One-Stage Domain Adaptation Network for Unsupervised Nighttime Semantic Segmentation

CPU inference engine that delivers unprecedented performance for sparse models

Repo for CReST: A Class-Rebalancing Self-Training Framework for Imbalanced Semi-Supervised Learning

Web service for facial landmark detection, head pose estimation, facial action unit recognition, and eye-gaze estimation based on OpenFace 2.0

Project ArXiv Citation Network

Research - dataset and code for 2016 paper Learning a Driving Simulator

Official repository for the paper "Going Beyond Linear Transformers with Recurrent Fast Weight Programmers"

Machine learning library for fast and efficient Gaussian mixture models

Moment-DETR code and QVHighlights dataset

CV backbones including GhostNet, TinyNet and TNT, developed by Huawei Noah's Ark Lab.

Contrastive Learning Inverts the Data Generating Process

Command-line tool for downloading and extending the RedCaps dataset.

YOLOv3 in PyTorch > ONNX > CoreML > TFLite

Latex code for making neural networks diagrams