Monocular, One-stage, Regression of Multiple 3D People
ROMP, accepted by ICCV 2021, is a concise one-stage network for multi-person 3D mesh recovery from a single image.
-
Simple. Concise one-stage framework for simultaneous person detection and 3D body mesh recovery.
-
Fast. ROMP can achieve real-time inference on a 1070Ti GPU.
-
Strong. ROMP achieves superior performance on multiple challenging multi-person/occlusion benchmarks.
-
Easy to use. We provide user friendly testing API and webcam demos.
Contact: [email protected]. Feel free to contact me for related questions or discussions! arXiv paper.
Table of contents
- Features
- News
- Getting Started
- Inference
- Train
- Evaluation
- Bugs report
- Citation
- Contributor
- Acknowledgement
Features
- Running the examples on Google Colab.
- Real-time online multi-person webcam demo for driving textured SMPL model. We also provide a wardrobe for changing clothes.
- Batch processing images/videos via command line / jupyter notebook / calling ROMP as a python lib.
- Exporting the captured single-person motion to FBX file for Blender/Unity usage.
- Training and evaluation for re-implementing our results presented in paper.
- Convenient API for 2D / 3D visualization, parsed datasets.
News
2021/12/2: Add optional renderers (pyrender or pytorch3D). Fix some bugs reported in issues.
2021/9/13: Low FPS / args parsing bugs are fixed. Support calling as a python lib.
2021/9/10: Training code release. API optimization.
Old logs
Getting started
Try on Google Colab
It allows you to run the project in the cloud, free of charge. Let's give the prepared Google Colab demo a try.
Installation
Please refer to install.md for installation.
Inference
Currently, we support processing images, video or real-time webcam.
Pelease refer to config_guide.md for configurations.
ROMP can be called as a python lib inside the python code, jupyter notebook, or from command line / scripts, please refer to Google Colab demo for examples.
Processing images
To re-implement the demo results, please run
cd ROMP
# change the `inputs` in configs/image.yml to /path/to/your/image folder, then run
sh scripts/image.sh
# or run the command like
python -m romp.predict.image --inputs=demo/images --output_dir=demo/image_results
Please refer to config_guide.md for saving the estimated mesh/Center maps/parameters dict.
For interactive visualization, please run
python -m romp.predict.image --inputs=demo/images --output_dir=demo/image_results --show_mesh_stand_on_image --interactive_vis
Caution: To use show_mesh_stand_on_image and interactive_vis, you must run ROMP on a computer with visual desktop to support the rendering. Most remote servers without visual desktop is not supported. Please use save_visualization_on_img instead.
Here, we show an example of calling ROMP as a python lib to process images.
click here to show the code
```bash
# set the absolute path to ROMP
path_to_romp = '/path/to/ROMP'
import os,sys
sys.path.append(path_to_romp)
# set the detailed configurations
from romp.lib.config import ConfigContext, parse_args, args
ConfigContext.parsed_args = parse_args(["--configs_yml=configs/image.yml",'--inputs=/path/to/images_folder', '--output_dir=/path/to/save/image_results', '--save_centermap', False]) # Be caution that setting the bool configs needs two elements, ['--config', True/False]
# import the ROMP image processor
from romp.predict.image import Image_processor
processor = Image_processor(args_set=args())
results_dict = processor.run(args().inputs) # you can change the args().inputs to other /path/to/images_folder
```
Processing videos
cd ROMP
python -m romp.predict.video --inputs=demo/videos/sample_video.mp4 --output_dir=demo/sample_video_results --save_visualization_on_img --save_dict_results
# or you can set all configurations in configs/video.yml, then run
sh scripts/video.sh
We notice that some users only want to extract the motion of the formost person, like this
python -m romp.predict.video --inputs=demo/videos/demo_video_frames --output_dir=demo/demo_video_fp_results --show_largest_person_only --save_dict_results --show_mesh_stand_on_image
All functions can be combined or work individually. Welcome to try them.
Here, we show an example of calling ROMP as a python lib to process videos.
click here to show the code
```bash
# set the absolute path to ROMP
path_to_romp = '/path/to/ROMP'
import os,sys
sys.path.append(path_to_romp)
# set the detailed configurations
from romp.lib.config import ConfigContext, parse_args, args
ConfigContext.parsed_args = parse_args(["--configs_yml=configs/video.yml",'--inputs=/path/to/video', '--output_dir=/path/to/save/video_results', '--save_visualization_on_img',False]) # Be caution that setting the bool configs needs two elements, ['--config', True/False]
# import the ROMP image processor
from romp.predict.video import Video_processor
processor = Video_processor(args_set=args())
results_dict = processor.run(args().inputs) # you can change the args().inputs to other /path/to/video
```
Webcam
To do this you just need to run:
cd ROMP
sh scripts/webcam.sh
To drive a character in Blender, please refer to expert.md.
Export
Export to Blender FBX
Please refer to expert.md to export the results to fbx files for Blender usage. Currently, this function only support the single-person video cases. Therefore, please test it with demo/videos/sample_video2_results/sample_video2.mp4, whose results would be saved to demo/videos/sample_video2_results.
Blender Addons
Chuanhang Yan : developing an addon for driving character in Blender.
VLT Media creates a QuickMocap-BlenderAddon to read the .npz file created by ROMP. Clean & smooth the resulting keyframes.
Train
Please prepare the training datasets following dataset.md, and then refer to train.md for training.
Evaluation
Please refer to evaluation.md for evaluation on benchmarks.
Bugs report
Please refer to bug.md for solutions. Welcome to submit the issues for related bugs. I will solve them as soon as possible.
Citation
@InProceedings{ROMP,
author = {Sun, Yu and Bao, Qian and Liu, Wu and Fu, Yili and Michael J., Black and Mei, Tao},
title = {Monocular, One-stage, Regression of Multiple 3D People},
booktitle = {ICCV},
month = {October},
year = {2021}
}
Contributor
This repository is currently maintained by Yu Sun.
ROMP has also benefited from many developers, including
- Marco Musy : help in the textured SMPL visualization.
- Gavin Gray : adding support for an elegant context manager to run code in a notebook.
- VLT Media : adding support for running on Windows & batch_videos.py.
- Chuanhang Yan : developing an addon for driving character in Blender.
Acknowledgement
We thank Peng Cheng for his constructive comments on Center map training.
Here are some great resources we benefit:
- SMPL models and layer is borrowed from MPII SMPL-X model.
- Some functions are borrowed from HMR-pytorch and SPIN.
- The evaluation code and GT annotations of 3DPW dataset is brought from 3dpw-eval and VIBE.
- 3D mesh visualization is supported by vedo, EasyMocap, minimal-hand, Open3D, and Pyrender.
Please consider citing their papers.
https://github.com/Arthur151/ROMP/issues/372#issuecomment-1345247684，看了这个回复，还是不太懂，如何实现
59 Dec 14, 2022
165 Nov 04, 2022
6 Dec 01, 2022
56 Dec 28, 2022
300 Dec 15, 2022
95 Dec 27, 2022
3 Jan 30, 2022
47 Nov 17, 2022
34 Jan 01, 2023
36 Jan 07, 2023
10 Nov 19, 2022
22 Dec 22, 2022
915 Jan 01, 2023
100 Sep 28, 2022
7 Jul 04, 2022
7.7k Jan 03, 2023
19 Jan 05, 2022
112 Aug 10, 2022
1 Jan 13, 2022
5 Nov 21, 2022