Sync2Gen
Code for ICCV 2021 paper: Scene Synthesis via Uncertainty-Driven Attribute Synchronization
0. Environment
Environment: python 3.6 and cuda 10.0 on Ubuntu 18.04
- Pytorch 1.4.0
- tensorflow 1.14.0 (for tensorboard)
1. Dataset
├──dataset_3dfront/
├──data
├── bedroom
├── 0_abs.npy
├── 0_rel.pkl
├── ...
├── living
├── 0_abs.npy
├── 0_rel.pkl
├── ...
├── train_bedroom.txt
├── train_living.txt
├── val_bedroom.txt
└── val_living.txt
See 3D-FRONT Dataset for dataset generation.
2. VAE
2.1 Generate scenes from random noises
Download the pretrained model from https://drive.google.com/file/d/1VKNlEdUj1RBUOjBaBxE5xQvfsZodVjam/view?usp=sharing
Sync2Gen
└── log
└── 3dfront
├── bedroom
│ └── vaef_lr0001_w00001_B64
│ ├── checkpoint_eval799.tar
│ └── pairs
└── living
└── vaef_lr0001_w00001_B64
├── checkpoint_eval799.tar
└── pairs
type='bedroom'; # or living
CUDA_VISIBLE_DEVICES=0 python ./test_sparse.py --type $type --log_dir ./log/3dfront/$type/vaef_lr0001_w00001_B64 --model_dict=model_scene_forward --max_parts=80 --num_class=20 --num_each_class=4 --batch_size=32 --variational --latent_dim 20 --abs_dim 16 --weight_kld 0.0001 --learning_rate 0.001 --use_dumped_pairs --dump_results --gen_from_noise --num_gen_from_noise 100
The predictions are dumped in ./dump/$type/vaef_lr0001_w00001_B64
2.2 Training
To train the network:
type='bedroom'; # or living
CUDA_VISIBLE_DEVICES=0 python ./train_sparse.py --data_path ./dataset_3dfront/data --type $type --log_dir ./log/3dfront/$type/vaef_lr0001_w00001_B64 --model_dict=model_scene_forward --max_parts=80 --num_class=20 --num_each_class=4 --batch_size=64 --variational --latent_dim 20 --abs_dim 16 --weight_kld 0.0001 --learning_rate 0.001
3. Bayesian optimization
cd optimization
3.1 Prior generation
See Prior generation.
3.2 Optimization
type=bedroom # or living;
bash opt.sh $type vaef_lr0001_w00001_B64 EXP_NAME
We use Pytorch-LBFGS for optimization.
3.3 Visualization
There is a simple visualization tool:
type=bedroom # or living
bash vis.sh $type vaef_lr0001_w00001_B64 EXP_NAME
The visualization is in ./vis. {i:04d}_2(3)d_pred.png is the initial prediction from VAE. {i:04d}_2(3)d_sync.png is the optimized layout after synchronization.
Acknowledgements
The repo is built based on:
We thank the authors for their great job.
Contact
If you have any questions, you can contact Haitao Yang (yanghtr [AT] outlook [DOT] com).
2.1k Jan 09, 2023
28 Dec 05, 2022
111 Jan 05, 2023
189 Dec 28, 2022
95 Dec 27, 2022
7 Sep 14, 2022
6 Jul 31, 2022
6 Jul 28, 2022
3 Dec 24, 2022
35 Nov 14, 2022
2 Dec 29, 2021
5 Dec 21, 2022
56 Nov 13, 2022
5 Nov 18, 2021
99 Jun 28, 2022
3 Aug 08, 2021
2.1k Dec 28, 2022
7 Nov 14, 2022
1.7k Jan 07, 2023
127 Dec 27, 2022