OpenMMLab 3D Human Parametric Model Toolbox and Benchmark

Implementation (integration) of the STAR model into MMHuman3D (resolves #148).

Summary

Copy and slightly adapt the original PyTorch implementation of the STAR model.

Changes

Compared to the original source code, I added explicit argument lists with their types. The model path can be loaded by specifying the directory where the gendered models are or by directly providing the path to the model.

Discussion

In contrast to SMPL model, at the moment we cannot simply import the STAR model and then potentially extend it, because the original repository does not properly expose STAR class to be imported. Therefore, it seems that the best solution was to copy and adapt the code.

The original license is now explicitly added as it feels somewhat improper to just copy the code, slightly adapt and then put another license. If we decide that it is OK, I will remove the second license.

Future Work (Body Models Refactoring)

I noticed that the extended SMPL classes abuse the original SMPL in several places, for example:

1. The override implementation of the forward() method returns a Python dictionary instead SMPLOutput or ModelOutput as it is done in the original class. This forces the default dictionary syntax, such as model['vertices'] instead of model.vertices. Also, it creates code redundancy that we see in several places where the outputs are recreated as dictionaries, such as here and here. The proper solution is to use an (extended) data class definition such as this.
2. The argument list for extended classes can be simplified. Instead of explicitly providing arguments, even repeating the arguments of the original method, the arguments can be kept as (*args, **kwargs) such as in SPIN or least we should avoid repeating the arguments from the original methods.
3. The keypoint (re)mapping logic should be moved from the body models implementations as the keypoints have nothing to do with the body models once they are extracted from the vertices.
4. The gendered models such as this seem redundant as the original SMPL model already accepts the gender parameter.
5. Improve unit tests (at least for the code coverage).

We can discuss these and other improvements in a separate issue.

@caizhongang why Intrinsic Matrix K defined as

K = [
        [fx,   0,   px,   0],
        [0,   fy,   py,   0],
        [0,    0,    0,   1],
        [0,    0,    1,   0],
    ]

rather than

K = [
        [fx,   0,   px],
        [0,   fy,   py],
        [0,    0,    1]
    ]

https://openmmlab-share.oss-cn-hangzhou.aliyuncs.com/mmhuman3d/models/hybrik/20211109_164017.log?versionId=CAEQHhiBgICdvonS6xciIDdiNGYzY2Q3N2NiMTQ5MzdhOTZjYjEwZDM0ZjI3ODU1

Thank you for your great work! I run the single-person demo according to page, and I tried hmr and vibe. the outputs are the same format. just like the following:

['__key_strict__', '__data_len__', '__keypoints_compressed__', 'smpl',
 'verts', 'pred_cams', 'bboxes_xyxy', 'image_path', 'person_id', 'frame_id']

I got the ouput which is correctly visualization, but I have some (maybe simple) questions: The input images are not square but rectangle, but the outputs pred_cams is of (frames_num,3) stand for s,tx,ty for each image. I guess the pred_cams is 'weak perspective camera parameters in cropped image space (s,tx,ty)' , but I don't know how to get the cropped images. By using the pred_cams (s,tx,ty) to calculate the extrinsics of original images seems not good. So I wander to know how to get cropped images? and how to get cams of (sx,sy,tx,ty)?

File "demo/estimate_smpl.py", line 296, in main(args) File "demo/estimate_smpl.py", line 195, in main verts, K0, img_index = single_person_with_mmdet(args, frames_iter) File "demo/estimate_smpl.py", line 65, in single_person_with_mmdet mesh_results = inference_model( File "/home/bixueting/code/mmhuman3d/mmhuman3d/apis/inference.py", line 139, in inference_model inference_pipeline = [LoadImage()] + cfg.inference_pipeline File "/home/bixueting/anaconda3/envs/mmhuman3d/lib/python3.8/site-packages/mmcv/utils/config.py", line 507, in getattr return getattr(self._cfg_dict, name) File "/home/bixueting/anaconda3/envs/mmhuman3d/lib/python3.8/site-packages/mmcv/utils/config.py", line 48, in getattr raise ex AttributeError: 'ConfigDict' object has no attribute 'inference_pipeline'

Please leave a comment on what new method/feature/dataset you would like us to add in MMHuman3D.

The most upvoted suggestions will be prioritized!

Update: we have moved our Wishlist to this discussion. See you there!

Demo Command:

python demo/etimate_smpl_video.py configs/hmr/resnet50_hmr_pw3d.py data/checkpoints/resnet50_hmr_pw3d-04f40f58_20211201.pth --single_person_demo --det_config demo/mmdetection_cfg/faster_rcnn_r50_fpn_coco.py --det_checkpoint data/checkpoints/detection/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth --input_path work_dir/video.mp4 --output work_dir/results/ --show_path work_dir/results/ --smooth_type savgol --draw_bbox

Get the following error @inference:

  File "demo/etimate_smpl_video.py", line 337, in <module>
    main(args)
  File "demo/etimate_smpl_video.py", line 241, in main
    single_person_with_mmdet(args, frames_iter)
  File "demo/etimate_smpl_video.py", line 91, in single_person_with_mmdet
    mesh_results = inference_video_based_model(
  File "mmhuman3d/mmhuman3d/apis/inference.py", line 244, in inference_video_based_model
    feature_sequences = _collate_feature_sequence(input_features,
  File "mmhuman3d/apis/inference.py", line 465, in _collate_feature_sequence
    C = extracted_features[target_frame][0]['features'].shape[0]
KeyError: 'features'

Same error for SPIN

P. S. The the config for both hmr & spin is missing data.test.seq_len.

Resolves #67

• Inplace renaming operation to obtain correct image path in agora converter
• Use tuple instead of string check in agora converter
• Retrieve correct image path using id in posetrack converter
• Fix inconsistency is cv2.inwrite image path and stored image path in surreal converter
• Fix H36M cam_param to store a dictionary of camera parameters

Resolves #70

• Fix type in instances where bbox_xywh is bbox_xyxy
• Split _bbox_expand to bbox_expand (only bbox expansion) and xyxy2xywh (xyxy conversion to xywh)

Resolves #79 issue of keypoints2d_loss not converging

• Make a copy of keypoints2d instead of directly modifying when applying augmentations

Resolves #49 .

I want to convert Human36M dataset and I use the following script

python tools/convert_datasets.py --datasets h36m_p1 --root_path ../data/datasets/ --output_path ../data/processed_datasets

Then I get the error "cam_param is absent in HumanData.SUPPORTED_KEYS."

File "/root/anaconda3/envs/zxh/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 528, in forward self.reducer.prepare_for_backward([]) RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing loss. You can enable unused parameter detection by (1) passing the keyword argument find_unused_parameters=True to torch.nn.parallel.DistributedDataParallel; (2) making sure all forward function outputs participate in calculating loss. If you already have done the above two steps, then the distributed data parallel module wasn't able to locate the output tensors in the return value of your module's forward function. Please include the loss function and the structure of the return value of forward of your module when reporting this issue (e.g. list, dict, iterable). Traceback (most recent call last):

THX!

Refer to

• [CI] Cancel previous runs that are not completed
• Skip CI when some specific files were changed
• Use the official markdownlint hook
• Add codespell hook

I install torch1.13.0+cuda116 and mmcv using:

conda install pytorch==1.13.0 torchvision pytorch-cuda=11.6 -c pytorch -c nvidia
pip install "mmcv-full>=1.3.17,<=1.5.3" -f https://download.openmmlab.com/mmcv/dist/cu116/torch1.13/index.html

but I got error

Traceback (most recent call last):
  File "/home/my/Pose/mmhuman3d/tools/train.py", line 188, in <module>
    main()
  File "/home/my/Pose/mmhuman3d/tools/train.py", line 135, in main
    env_info_dict = collect_env()
  File "/home/my/Pose/mmhuman3d/mmhuman3d/utils/collect_env.py", line 9, in collect_env
    env_info = collect_base_env()
  File "/home/my/anaconda3/envs/mmhuman3d_torch113/lib/python3.9/site-packages/mmcv/utils/env.py", line 112, in collect_env
    from mmcv.ops import get_compiler_version, get_compiling_cuda_version
  File "/home/my/anaconda3/envs/mmhuman3d_torch113/lib/python3.9/site-packages/mmcv/ops/__init__.py", line 2, in <module>
    from .active_rotated_filter import active_rotated_filter
  File "/home/my/anaconda3/envs/mmhuman3d_torch113/lib/python3.9/site-packages/mmcv/ops/active_rotated_filter.py", line 10, in <module>
    ext_module = ext_loader.load_ext(
  File "/home/my/anaconda3/envs/mmhuman3d_torch113/lib/python3.9/site-packages/mmcv/utils/ext_loader.py", line 13, in load_ext
    ext = importlib.import_module('mmcv.' + name)
  File "/home/my/anaconda3/envs/mmhuman3d_torch113/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
ImportError: /home/my/anaconda3/envs/mmhuman3d_torch113/lib/python3.9/site-packages/mmcv/_ext.cpython-39-x86_64-linux-gnu.so: undefined symbol: _ZN2at6mm_outERNS_6TensorERKS0_S3_

This PR mainly focuses on adding a new data structure for multi-human training.

• [x] Add a new data structure for multi-human training
• [x] Update the corresponding document
• [x] Update unitest

Note: This PR shouldn't be merged until #257 and #264 are merged.

I am having trouble getting mmhuman3d to install on Ubuntu 18.04 with an RTX 3070 GPU using conda.

I am trying to execute

pip install -v -e .

as described in the installation documentation. I followed every step despite using newer versions for Pytorch (1.13) and CUDA (11.6) and Torchvision as these changes are necessary due to RTX 30xx compatiblity issues.

However, setup fails and this error is thrown: /home/nico90/anaconda3/envs/open-mmlab/lib/python3.8/site-packages/setuptools/command/easy_install.py:144: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools. warnings.warn( /home/nico90/anaconda3/envs/open-mmlab/lib/python3.8/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools. warnings.warn( Traceback (most recent call last): File "", line 2, in File "", line 34, in File "/home/nico90/mmhuman3d/setup.py", line 117, in setup( File "/home/nico90/anaconda3/envs/open-mmlab/lib/python3.8/site-packages/setuptools/init.py", line 87, in setup return distutils.core.setup(**attrs) File "/home/nico90/anaconda3/envs/open-mmlab/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 185, in setup return run_commands(dist) File "/home/nico90/anaconda3/envs/open-mmlab/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 201, in run_commands dist.run_commands() File "/home/nico90/anaconda3/envs/open-mmlab/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 968, in run_commands self.run_command(cmd) File "/home/nico90/anaconda3/envs/open-mmlab/lib/python3.8/site-packages/setuptools/dist.py", line 1217, in run_command super().run_command(command) File "/home/nico90/anaconda3/envs/open-mmlab/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 987, in run_command cmd_obj.run() File "/home/nico90/anaconda3/envs/open-mmlab/lib/python3.8/site-packages/setuptools/command/develop.py", line 34, in run self.install_for_development() File "/home/nico90/anaconda3/envs/open-mmlab/lib/python3.8/site-packages/setuptools/command/develop.py", line 114, in install_for_development self.run_command('build_ext') File "/home/nico90/anaconda3/envs/open-mmlab/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 319, in run_command self.distribution.run_command(command) File "/home/nico90/anaconda3/envs/open-mmlab/lib/python3.8/site-packages/setuptools/dist.py", line 1217, in run_command super().run_command(command) File "/home/nico90/anaconda3/envs/open-mmlab/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 987, in run_command cmd_obj.run() File "/home/nico90/anaconda3/envs/open-mmlab/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 84, in run _build_ext.run(self) File "/home/nico90/anaconda3/envs/open-mmlab/lib/python3.8/site-packages/Cython/Distutils/old_build_ext.py", line 186, in run _build_ext.build_ext.run(self) File "/home/nico90/anaconda3/envs/open-mmlab/lib/python3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 346, in run self.build_extensions() File "/home/nico90/anaconda3/envs/open-mmlab/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 372, in build_extensions build_ext.build_extensions(self) File "/home/nico90/anaconda3/envs/open-mmlab/lib/python3.8/site-packages/Cython/Distutils/old_build_ext.py", line 195, in build_extensions _build_ext.build_ext.build_extensions(self) File "/home/nico90/anaconda3/envs/open-mmlab/lib/python3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 466, in build_extensions self._build_extensions_serial() File "/home/nico90/anaconda3/envs/open-mmlab/lib/python3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 492, in _build_extensions_serial self.build_extension(ext) File "/home/nico90/anaconda3/envs/open-mmlab/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 246, in build_extension _build_ext.build_extension(self, ext) File "/home/nico90/anaconda3/envs/open-mmlab/lib/python3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 547, in build_extension objects = self.compiler.compile( File "/home/nico90/anaconda3/envs/open-mmlab/lib/python3.8/site-packages/setuptools/_distutils/ccompiler.py", line 599, in compile self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts) File "/home/nico90/anaconda3/envs/open-mmlab/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 288, in unix_wrap_compile "'-fPIC'"] + cflags + _get_cuda_arch_flags(cflags) File "/home/nico90/anaconda3/envs/open-mmlab/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1027, in _get_cuda_arch_flags raise ValueError("Unknown CUDA arch ({}) or GPU not supported".format(arch)) ValueError: Unknown CUDA arch (8.6) or GPU not supported error: subprocess-exited-with-error

× python setup.py develop did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
full command: /home/nico90/anaconda3/envs/open-mmlab/bin/python -c '
exec(compile('"'"''"'"''"'"'
# This is <pip-setuptools-caller> -- a caller that pip uses to run setup.py
#
# - It imports setuptools before invoking setup.py, to enable projects that directly
#   import from `distutils.core` to work with newer packaging standards.
# - It provides a clear error message when setuptools is not installed.
# - It sets `sys.argv[0]` to the underlying `setup.py`, when invoking `setup.py` so
#   setuptools doesn'"'"'t think the script is `-c`. This avoids the following warning:
#     manifest_maker: standard file '"'"'-c'"'"' not found".
# - It generates a shim setup.py, for handling setup.cfg-only projects.
import os, sys, tokenize

try:
    import setuptools
except ImportError as error:
    print(
        "ERROR: Can not execute `setup.py` since setuptools is not available in "
        "the build environment.",
        file=sys.stderr,
    )
    sys.exit(1)

__file__ = %r
sys.argv[0] = __file__

if os.path.exists(__file__):
    filename = __file__
    with tokenize.open(__file__) as f:
        setup_py_code = f.read()
else:
    filename = "<auto-generated setuptools caller>"
    setup_py_code = "from setuptools import setup; setup()"

exec(compile(setup_py_code, filename, "exec"))
'"'"''"'"''"'"' % ('"'"'/home/nico90/mmhuman3d/setup.py'"'"',), "<pip-setuptools-caller>", "exec"))' develop --no-deps
cwd: /home/nico90/mmhuman3d/

error: subprocess-exited-with-error

× python setup.py develop did not run successfully. │ exit code: 1 ╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

torch.cuda.is_available() is returning True

Does anybody have a fix at hand?

Please write your suggestions in below comments. I will rearrange them and summarize a plan list.

Cameras.

• [ ] Add verbose switch in renderer.

Renderer.

• [ ] Add DecoMR uv render.
• [ ] Simplify visualize_smpl.
• [ ] Accelerate visualize_smpl.
• [ ] Add all the tutorials for different datasets or algorithms.

The PR is now open. #296