Robot reinforcement learning - transferring end-to-end videomotor control from simulation to realworld (curl 2017)

6.1 brief introduction

Scene oriented ： End to end visual drive （ Images -> Torque or torque ,Visuomotor）、 Multistage tasks （multi-stage task）.

Mission ： Locate the square object 、reach、grasp、 Position the basket 、 Put the cubes in the basket

The method mainly includes two steps ：（1） Calculate and collect trajectories in the simulation environment , That is to control the speed ;（2） Training CNN, Learn the mapping from image to speed , Use Domain randomization enhanced .

Presentation data ： By solving inverse kinematics in Cartesian coordinate system , first person Images .

The Internet ： Input Image and joint angle , Output Motor speed （ adopt PID Make the joint reach this speed ）. Auxiliary output The position of the block and the manipulator when , Network performance will be improved .

experiment Demonstrated under dynamic lighting conditions 、 There are interferences 、 Objects move The effect of the scene .

6.2 Method

1、 How to implement multi-stage tasks ？ Does the network know the current task progress ？

This is not explicitly stated in the paper , It is presumed that ：LSTM Network input four consecutive frames of images , Learn the current task progress according to the image changes ;

Such as ：

（1） When the manipulator is empty , The current task goal of the network is Move the manipulator over the block

（2） The manipulator is moving for four consecutive frames , When the manipulator is above the block in the current frame , The current task goal of the network is Closing manipulator

（3） When the block is in the manipulator , The current task goal of the network is Move the manipulator over the basket

（4） When the manipulator moves to the top of the basket , The current task goal of the network is Open the manipulator

6.3 idea

6.3.1 Paper questions

1、 Behavioral cloning cannot handle scenes you have never seen , Therefore need 100 10000 images ; If you use behavioral cloning training first , Then use reinforcement learning , Fewer samples may be required ？

2、 Model too fixed , If I want to grab a circular object 、 Put it in another basket , Need to retrain the network ; You can try to add the task target to the input of the network

6.3.2 idea

1、 Behavioral cloning + Reinforcement learning

2、 Target the task （ Such as object image 、 Map nodes, etc ） Input added to the network , So that the method can deal with different objects

3、 The network input can contain The state of the manipulator , similar QT-Opt;

4、 You can draw lessons from Network output joint speed 、 Domain randomization 、LSTM Learning task status and progress .

6.4 Notes on the original paper

pdf Download address ：https://download.csdn.net/download/qq_40081208/85788235