Minimal Hand
A minimal solution to hand motion capture from a single color camera at over 100fps. Easy to use, plug to run.
This project provides the core components for hand motion capture:
- estimating joint locations from a monocular RGB image (DetNet)
- estimating joint rotations from locations (IKNet)
We focus on:
- ease of use (all you need is a webcam)
- time efficiency (on our 1080Ti, 8.9ms for DetNet, 0.9ms for IKNet)
- robustness to occlusion, hand-object interaction, fast motion, changing scale and view point
Some links: [video] [paper] [supp doc] [webpage]
The author is too busy to collect the training code for release. On the other hand, it should not be difficult to implement the training part. Feel free to open an issue for any encountered problems.
Pytorch Version
Here is a pytorch version implemented by @MengHao666. I didn't personally check it but I believe it worth trying. Many thanks to @MengHao666 !
With Unity
Here is a project that connects this repo to unity. It looks very cool and many thanks to @vinnik-dmitry07 !
Usage
Install dependencies
Please check requirements.txt. All dependencies are available via pip and conda.
Prepare MANO hand model
- Download MANO model from here and unzip it.
- In
config.py, setOFFICIAL_MANO_PATHto the left hand model. - Run
python prepare_mano.py, you will get the converted MANO model that is compatible with this project atconfig.HAND_MESH_MODEL_PATH.
Prepare pre-trained network models
- Download models from here.
- Put
detnet.ckpt.*inmodel/detnet, andiknet.ckpt.*inmodel/iknet. - Check
config.py, make sure all required files are there.
Run the demo for webcam input
python app.py- Put your right hand in front of the camera. The pre-trained model is for left hand, but the input would be flipped internally.
- Press
ESCto quit. - Although the model is robust to variant scales, most ideally the image should be 1.3x larger than the hand bounding box. A good bounding box may result in better accuracy. You can track the bounding box with the 2D predictions of the model.
We found that the model may fail on some "simple" poses. We think this is because such poses were no presented in the training data. We are working on a v2 version with further extended data to tackle this problem.
Use the models in your project
Please check wrappers.py.
IKNet Alternative
We also provide an optimization-based IK solver here.
Dataset
The detection model is trained with following datasets:
The IK model is trained with the poses shipped with MANO.
Citation
This is the official implementation of the paper "Monocular Real-time Hand Shape and Motion Capture using Multi-modal Data" (CVPR 2020).
The quantitative numbers reported in the paper can be found in plot.py.
If you find the project helpful, please consider citing us:
@inproceedings{zhou2020monocular,
title={Monocular Real-time Hand Shape and Motion Capture using Multi-modal Data},
author={Zhou, Yuxiao and Habermann, Marc and Xu, Weipeng and Habibie, Ikhsanul and Theobalt, Christian and Xu, Feng},
booktitle={Proceedings of the IEEE International Conference on Computer Vision},
pages={0--0},
year={2020}
}
(2) mpii_ref hand in rotated xyz view, showing unnaturally curved fingers and very long wrist-to-thumb connection
(3) For comparison: mediapipe hand in front view
(4) For comparison: mediapipe hand in same xyz view as above
9 Nov 06, 2022
41 Dec 14, 2022
49 Dec 20, 2022
269 Jan 05, 2023
109 Nov 29, 2022
3 Oct 31, 2022
41 Jan 16, 2022
381 Dec 30, 2022
25 Jul 27, 2022
27 Dec 29, 2022
1.3k Dec 28, 2022
981 Dec 29, 2022
2.8k Dec 27, 2022
33 Dec 05, 2022
72 Dec 23, 2022
11 Nov 28, 2022
91 Dec 30, 2022
45 Jul 22, 2022
131 Jan 07, 2023
8 Oct 08, 2022