Sync2Gen Code for ICCV 2021 paper: Scene Synthesis via Uncertainty-Driven Attribute Synchronization

Last update: Dec 30, 2022

Related tags

Deep Learning Sync2Gen

Overview

Sync2Gen

Code for ICCV 2021 paper: Scene Synthesis via Uncertainty-Driven Attribute Synchronization

0. Environment

Environment: python 3.6 and cuda 10.0 on Ubuntu 18.04

Pytorch 1.4.0
tensorflow 1.14.0 (for tensorboard)

1. Dataset

├──dataset_3dfront/
    ├──data
        ├── bedroom
            ├── 0_abs.npy
            ├── 0_rel.pkl
            ├── ...
        ├── living
            ├── 0_abs.npy
            ├── 0_rel.pkl
            ├── ...
        ├── train_bedroom.txt
        ├── train_living.txt
        ├── val_bedroom.txt
        └── val_living.txt

See 3D-FRONT Dataset for dataset generation.

2. VAE

2.1 Generate scenes from random noises

Download the pretrained model from https://drive.google.com/file/d/1VKNlEdUj1RBUOjBaBxE5xQvfsZodVjam/view?usp=sharing

Sync2Gen
└── log
    └── 3dfront
        ├── bedroom
        │   └── vaef_lr0001_w00001_B64
        │       ├── checkpoint_eval799.tar
        │       └── pairs
        └── living
            └── vaef_lr0001_w00001_B64
                ├── checkpoint_eval799.tar
                └── pairs

type='bedroom'; # or living
CUDA_VISIBLE_DEVICES=0 python ./test_sparse.py  --type $type  --log_dir ./log/3dfront/$type/vaef_lr0001_w00001_B64 --model_dict=model_scene_forward --max_parts=80 --num_class=20 --num_each_class=4 --batch_size=32 --variational --latent_dim 20 --abs_dim 16  --weight_kld 0.0001  --learning_rate 0.001 --use_dumped_pairs --dump_results --gen_from_noise --num_gen_from_noise 100

The predictions are dumped in ./dump/$type/vaef_lr0001_w00001_B64

2.2 Training

To train the network:

type='bedroom'; # or living
CUDA_VISIBLE_DEVICES=0 python ./train_sparse.py --data_path ./dataset_3dfront/data  --type $type  --log_dir ./log/3dfront/$type/vaef_lr0001_w00001_B64  --model_dict=model_scene_forward --max_parts=80 --num_class=20 --num_each_class=4 --batch_size=64 --variational --latent_dim 20 --abs_dim 16  --weight_kld 0.0001  --learning_rate 0.001

3. Bayesian optimization

cd optimization

3.1 Prior generation

See Prior generation.

3.2 Optimization

type=bedroom # or living;
bash opt.sh $type vaef_lr0001_w00001_B64  EXP_NAME

We use Pytorch-LBFGS for optimization.

3.3 Visualization

There is a simple visualization tool:

type=bedroom # or living
bash vis.sh $type vaef_lr0001_w00001_B64 EXP_NAME

The visualization is in ./vis. {i:04d}_2(3)d_pred.png is the initial prediction from VAE. {i:04d}_2(3)d_sync.png is the optimized layout after synchronization.

Acknowledgements

The repo is built based on:

We thank the authors for their great job.

Contact

If you have any questions, you can contact Haitao Yang (yanghtr [AT] outlook [DOT] com).

Sync2Gen Code for ICCV 2021 paper: Scene Synthesis via Uncertainty-Driven Attribute Synchronization

Related tags

Overview

Sync2Gen

0. Environment

1. Dataset

2. VAE

2.1 Generate scenes from random noises

2.2 Training

3. Bayesian optimization

3.1 Prior generation

3.2 Optimization

3.3 Visualization

Acknowledgements

Contact

Owner

Haitao Yang

Implementation of a memory efficient multi-head attention as proposed in the paper, "Self-attention Does Not Need O(n²) Memory"

reimpliment of DFANet: Deep Feature Aggregation for Real-Time Semantic Segmentation

Official PyTorch implementation of Learning Intra-Batch Connections for Deep Metric Learning (ICML 2021) published at International Conference on Machine Learning

Turning pixels into virtual points for multimodal 3D object detection.

Implementation of RegretNet with Pytorch

Pytorch implementation AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks

Physics-informed convolutional-recurrent neural networks for solving spatiotemporal PDEs

YOLO5Face: Why Reinventing a Face Detector (https://arxiv.org/abs/2105.12931)

Semi-supervised Implicit Scene Completion from Sparse LiDAR

Companion code for the paper Theoretical characterization of uncertainty in high-dimensional linear classification

Implicit Deep Adaptive Design (iDAD)

Open Source Differentiable Computer Vision Library for PyTorch

Data cleaning, missing value handle, EDA use in this project

This repo holds code for TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation

Code of the paper "Performance-Efficiency Trade-offs in Unsupervised Pre-training for Speech Recognition"

Estimating Example Difficulty using Variance of Gradients

MISSFormer: An Effective Medical Image Segmentation Transformer

Generic Event Boundary Detection: A Benchmark for Event Segmentation

PyTorch code for our ECCV 2018 paper "Image Super-Resolution Using Very Deep Residual Channel Attention Networks"

Decorators for maximizing memory utilization with PyTorch & CUDA