An OpenAI-Gym Package for Training and Testing Reinforcement Learning algorithms with OpenSim Models

Overview

Logo

Authors: Utkarsh A. Mishra and Dr. Dimitar Stanev

Advisors: Dr. Dimitar Stanev and Prof. Auke Ijspeert, Biorobotics Laboratory (BioRob), EPFL

Video Playlist: https://www.youtube.com/playlist?list=PLDvnH871wUkFPOcCKcsTN6ZzzjNZOVlt_

The bioimiitation-gym package is a python package that provides a gym environment for training and testing OpenSim models. The gym environment is based on the OpenAI gym package.

This work is towards a framework aimed towards learning to imitate human gaits. Humans exhibit movements like walking, running, and jumping in the most efficient manner, which served as the source of motivation for this project. Skeletal and Musculoskeletal human models were considered for motions in the sagittal and frontal plane, and results from both were compared exhaustively. While skeletal models are driven with motor actuation, musculoskeletal models perform through muscle-tendon actuation.

Baseline Architecture

Model-free reinforcement learning algorithms were used to optimize inverse dynamics control actions to satisfy the objective of imitating a reference motion along with secondary objectives of minimizing effort in terms of power spent by motors and metabolic energy consumed by the muscles. On the one hand, the control actions for the motor actuated model is the target joint angles converted into joint torques through a Proportional-Differential controller. While on the other hand, the control actions for the muscle-tendon actuated model is the muscle excitations converted implicitly to muscle activations and then to muscle forces which apply moments on joints. Muscle-tendon actuated models were found to have superiority over motor actuation as they are inherently smooth due to muscle activation dynamics and don't need any external regularizers.

Results

All the results and analysis are presented in an illustrative, qualitative, and quantitative manner.

Installation

Please follow the instructions in the installation.md file to install the package.

Environment in the bioimitation-gym package

All environments in the bioimitation-gym package are provided in the biomitation_envs/imitation_envs/envs directory. They are majorly divided into two categories:

  • muscle environments: These are the environments that are used for training the muscle tendon unit actuated model.
  • torque environments: These are the environments that are used for training the torque actuate model.

Further, 2D / planar and 3D / spatial environments are provided for each category. The tasks covered in each of the sub-categories are as follows:

  • Walking
  • Running
  • Jumping
  • Prosthetic Walking with a locked knee joint for the left leg
  • Walking with a typical Cerebel Palsy defect

The following 2D muscle actuated environment names can be used based on the package:

  • MuscleWalkingImitation2D-v0
  • MuscleRunningImitation2D-v0
  • MuscleJumpingImitation2D-v0
  • MuscleLockedKneeImitation2D-v0

The following 3D muscle actuated environment names can be used based on the package:

  • MuscleWalkingImitation3D-v0
  • MuscleRunningImitation3D-v0
  • MuscleJumpingImitation3D-v0
  • MuscleLockedKneeImitation3D-v0
  • MusclePalsyImitation3D-v0

The following 2D torque actuated environment names can be used based on the package:

  • TorqueWalkingImitation2D-v0
  • TorqueRunningImitation2D-v0
  • TorqueJumpingImitation2D-v0
  • TorqueLockedKneeImitation2D-v0

The following 3D torque actuated environment names can be used based on the package:

  • TorqueWalkingImitation3D-v0
  • TorqueRunningImitation3D-v0
  • TorqueJumpingImitation3D-v0
  • TorqueLockedKneeImitation3D-v0

Usage Instructions

The complete bioimitation directory consists of the following sub-directories:

  • imitation_envs: This directory contains the data and environments associated with the package.
  • learning_algorithm: This directory contains the learning algorithm used for several experiments. The code is the modified version of original SAC algorithm and is taken from the open source implementation of ikostrikov/jaxrl.

More information on the subdirectories can be found in their respective README files (if any).

The package is mostly based on the highly scalable and distributed reinforcement learning framework Ray RLLIB. The template scipts to train and test the models are provided in the tests directory.

To run a RLLIB training script, run the following command:

python tests/sample_rllib_training.py  --env_name MuscleWalkingImitation2D-v0

You can change the algorithm configurations in the configs directory. The configs/train_default.py file contains the default configuration for the train script and the configs/test_default.py file contains the default configuration for the test script which is:

python tests/sample_rllib_testing.py

The default environment configuration is provided in the configs/env_default.py file. Feel free to change the default configuration as per your needs. A typical script to test the environment is provided in the biomitation_envs/imitation_envs/envs directory is:

import os
from absl import app, flags
from ml_collections import config_flags
import gym
import bioimitation

FLAGS = flags.FLAGS

flags.DEFINE_string('env_name', 'MuscleWalkingImitation2D-v0', 'Name of the environment.')

config_flags.DEFINE_config_file(
    'config',
    'configs/env_default.py',
    'File path to the environment configuration.',
    lock_config=False)

def main(_):

    example_config = dict(FLAGS.config)

    env = gym.make(FLAGS.env_name, config=example_config)

    env.reset()

    for i in range(1000):
        _, _, done, _ = env.step(env.action_space.sample())
        if done:
            env.reset()

if __name__ == '__main__':
    app.run(main)

Don't forget to import the bioimitation package before running the script.

Citation

If you use this work in your research, please cite the following as:

@misc{
    mishra2021bioimitation,
    title = {BioImitation-Gym: A OpenAI-Gym Package for Training and Testing Reinforcement Learning algorithms with OpenSim Models},
    author = {Utkarsh A. Mishra and Dimitar Stanev and Auke Ijspeert},
    year = {2021},
    url = {https://github.com/UtkarshMishra/bioimitation-gym}
}
@article{mishra2021learning,
  title={Learning Control Policies for Imitating Human Gaits},
  author={Utkarsh A. Mishra},
  journal={arXiv preprint arXiv:2106.15273},
  year={2021}
}

References

[1] OsimRL project: https://osim-rl.kidzinski.com/

[2] OpenSim: https://github.com/opensim-org/opensim-core and https://opensim.stanford.edu/

[3] OpenAI Gym: https://gym.openai.com/

[4] Ray RLLIB: https://ray.readthedocs.io/en/latest/

[6] ikostrikov/jaxrl: https://github.com/ikostrikov/jaxrl

Owner
Utkarsh Mishra
Graduate from @iitroorkee (Batch of 2021), programming enthusiast. Reinforcement Learning, Robotics & Self-Driving interests me.
Utkarsh Mishra
This repository contains the code for "SBEVNet: End-to-End Deep Stereo Layout Estimation" paper by Divam Gupta, Wei Pu, Trenton Tabor, Jeff Schneider

SBEVNet: End-to-End Deep Stereo Layout Estimation This repository contains the code for "SBEVNet: End-to-End Deep Stereo Layout Estimation" paper by D

Divam Gupta 19 Dec 17, 2022
Yas CRNN model training - Yet Another Genshin Impact Scanner

Yas-Train Yet Another Genshin Impact Scanner 又一个原神圣遗物导出器 介绍 该仓库为 Yas 的模型训练程序 相关资料 MobileNetV3 CRNN 使用 假设你会设置基本的pytorch环境。 生成数据集 python main.py gen 训练

wormtql 18 Jan 08, 2023
Alpha-IoU: A Family of Power Intersection over Union Losses for Bounding Box Regression

Alpha-IoU: A Family of Power Intersection over Union Losses for Bounding Box Regression YOLOv5 with alpha-IoU losses implemented in PyTorch. Example r

Jacobi(Jiabo He) 147 Dec 05, 2022
PyJokes - Joking around with Python library pyjokes

Hi, it's Muhaimin again 👋 This is something unorthodox but cool. Don't forget t

Muhaimin A. Salay Kanton 1 Feb 02, 2022
This code is for eCaReNet: explainable Cancer Relapse Prediction Network.

eCaReNet This code is for eCaReNet: explainable Cancer Relapse Prediction Network. (Towards Explainable End-to-End Prostate Cancer Relapse Prediction

Institute of Medical Systems Biology 2 Jul 28, 2022
Offcial repository for the IEEE ICRA 2021 paper Auto-Tuned Sim-to-Real Transfer.

Offcial repository for the IEEE ICRA 2021 paper Auto-Tuned Sim-to-Real Transfer.

47 Jun 30, 2022
Pytorch implementation of the AAAI 2022 paper "Cross-Domain Empirical Risk Minimization for Unbiased Long-tailed Classification"

[AAAI22] Cross-Domain Empirical Risk Minimization for Unbiased Long-tailed Classification We point out the overlooked unbiasedness in long-tailed clas

PatatiPatata 28 Oct 18, 2022
[AI6101] Introduction to AI & AI Ethics is a core course of MSAI, SCSE, NTU, Singapore

[AI6101] Introduction to AI & AI Ethics is a core course of MSAI, SCSE, NTU, Singapore. The repository corresponds to the AI6101 of Semester 1, AY2021-2022, starting from 08/2021. The instructors of

AccSrd 1 Sep 22, 2022
Curved Projection Reformation

Description Assuming that we already know the image of the centerline, we want the lumen to be displayed on a plane, which requires curved projection

夜听残荷 5 Sep 11, 2022
[ICLR'19] Trellis Networks for Sequence Modeling

TrellisNet for Sequence Modeling This repository contains the experiments done in paper Trellis Networks for Sequence Modeling by Shaojie Bai, J. Zico

CMU Locus Lab 460 Oct 13, 2022
Implementation of Heterogeneous Graph Attention Network

HetGAN Implementation of Heterogeneous Graph Attention Network This is the code repository of paper "Prediction of Metro Ridership During the COVID-19

5 Dec 28, 2021
SoK: Vehicle Orientation Representations for Deep Rotation Estimation

SoK: Vehicle Orientation Representations for Deep Rotation Estimation Raymond H. Tu, Siyuan Peng, Valdimir Leung, Richard Gao, Jerry Lan This is the o

FIRE Capital One Machine Learning of the University of Maryland 12 Oct 07, 2022
When are Iterative GPs Numerically Accurate?

When are Iterative GPs Numerically Accurate? This is a code repository for the paper "When are Iterative GPs Numerically Accurate?" by Wesley Maddox,

Wesley Maddox 1 Jan 06, 2022
Public repository containing materials used for Feed Forward (FF) Neural Networks article.

Art041_NN_Feed_Forward Public repository containing materials used for Feed Forward (FF) Neural Networks article. -- Illustration of a very simple Fee

SolClover 2 Dec 29, 2021
Gym-TORCS is the reinforcement learning (RL) environment in TORCS domain with OpenAI-gym-like interface.

Gym-TORCS Gym-TORCS is the reinforcement learning (RL) environment in TORCS domain with OpenAI-gym-like interface. TORCS is the open-rource realistic

naoto yoshida 400 Dec 27, 2022
Provably Rare Gem Miner.

Provably Rare Gem Miner just another random project by yoyoismee.eth useful link main site market contract useful thing you should know read contract

34 Nov 22, 2022
Collaborative forensic timeline analysis

Timesketch Table of Contents About Timesketch Getting started Community Contributing About Timesketch Timesketch is an open-source tool for collaborat

Google 2.1k Dec 28, 2022
Security evaluation module with onnx, pytorch, and SecML.

🚀 🐼 🔥 PandaVision Integrate and automate security evaluations with onnx, pytorch, and SecML! Installation Starting the server without Docker If you

Maura Pintor 11 Apr 12, 2022
Controlling the MicriSpotAI robot from scratch

Abstract: The SpotMicroAI project is designed to be a low cost, easily built quadruped robot. The design is roughly based off of Boston Dynamics quadr

Florian Wilk 405 Jan 05, 2023
Matthew Colbrook 1 Apr 08, 2022