PRIMAL: Pathfinding via Reinforcement and Imitation Multi-Agent Learning Code Analysis

论文名称：PRIMAL: Pathfinding via Reinforcement and Imitation Multi-Agent Learning
论文地址：https://arxiv.org/abs/1809.03531
代码地址：https://github.com/gsartoretti/PRIMAL
相关链接：Research on the dynamic obstacle avoidance strategy of unmanned driving | Robot dynamic obstacle avoidance strategy | 行人轨迹预测 | 机器人导航,Three papers on reinforcement learning for multi-agent path planning

1. 准备工作

1.1 配置环境：

1. Create a new environment first：
   conda create --name PRIMAL python=3.6
2. 新建一个requirements.txt文件,Put all the packages you want to install in there,再运行 pip install -r requirements安装包：

Cython==0.28.4
gym==0.9.4
Tensorflow==1.13.0
numpy==1.16.0
matplotlib
imageio
tk
networkx

注：源代码中是python3.4 + tf 1.3 + numpy 1.13,但是因为我的pycharm版本较高,不能兼容低版本python,So I got itpython 3.6;py3.6无法兼容1.3版本的tensorflow,So I replaced it again1.13版本的tf.
3. 如果安装失败,You can also install them one by one（我采用的方法）
注：直接pip或者conda安装不了gym时,可以尝试以下两种方法：conda install -c conda-forge gym=0.9.4 或者 pip install gym -i https://pypi.tuna.tsinghua.edu.cn/simple
安装tensorflow出问题时,The following two sentences can be executed conda install cudatoolkit=10.0conda install tensorflow==1.13.1
4. 安装完成后,可以使用conda listcommand to check if both are installed：

1.2 调试代码

1. 安装readmeThe steps in the file come step by step：
   ① 命令行窗口进入cd od_mstar3文件夹,python setup.py build_ext --inplace;报错了error: Unable to find vcvarsall.bat
   网上搜了一下,发现是需要安装Visual Studio,And it needs to be checked during installation C++组件.如果安装时没有勾选,应当Reinstall in the toolbar,After restarting therevcvarsall模块了.参考文章：已安装vs2017 仍然报错Unable to find vcvarsall.bat

2. 然而,After installing it, it still gives an error o(╥﹏╥)o,参照：关于error: Unable to find vcvarsall.bat,我将msvc9compiler.py文件中的find_vcvarsall函数return改成 return r"C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Auxiliary\Build\vcvarsall.bat"
   其他参考文章：Unable to find vcvarsall.bat?

3. 改好vcvarsall.bat的bug后,又有了一个新bug： fatal error C1083: 无法打开包括文件: “boost/graph/graph_traits.hpp”: No such file or directory. 网上查了一下,这是C++里的BGL库,需要自己下载安装.It is used for pathfinding.Boost C++ Libraries 编译安装,oostDownload, install, compile, configure and use the guide（含Windows和Linux）
   ① 在官网中下载了 boost_1_53_.zip,解压运行bootstrap.bat的时候报错了 boost Compile-time error handling：Failed to build Boost.Build engine.查阅了很多资料,还是没解决,于是决定下载最新版本的boost工具包.
   Error again during installation,fatal error C1083: 无法打开包括文件: “corecrt.h,Windows 原生 cmd Compile under the window C++（cl命令）出现的问题及解决方法

   ② Installed by a friendVisual Studio 2022,can be installed smoothlyboost,So I also reinstalledvs2022.啊！终于可以啦！！

   boost工具安装教程,官方安装教程.

1.3 linux环境安装boost：

！！！！linux环境直接使用sudo apt-get install boost命令就可以安装了！ https://stackoverflow.com/questions/12578499/how-to-install-boost-on-ubuntu

2.代码解析：

2.1 setup.py

As requested by the author,This file should be run first~
setup.py是调用cpython,用 Python 的语法混合编写 Python 和 C/C++ 代码,提升 Python 速度
调用 C/C++ 代码 教程,Cython 基本用法 .

This code is runningcpython_od_matar.pyx这个文件,其中的find_path(world, init_pos, goals, inflation, time_limit)函数是使用ODrM算法进行探索.ODrMEquivalent to experts,generate a high-quality paths.
输入输出：

    world - matrix specifying obstacles, 1 for obstacle, 0 for free
    init_pos  - [[x, y], ...] specifying start position for each robot
    goals     - [[x, y], ...] specifying goal position for each robot
    inflation - inflation factor for heuristic
    time_limit - time until failure in seconds

    returns:
    [[[x1, y1], ...], [[x2, y2], ...], ...] path in the joint
    configuration space

2.2 A3C_RNN.py

因为pycharm在服务器运行jupyterThe file is a bit cumbersome,我将DRLMAPF_A3C_RNN.ipynb文件改成了A3C_RNN.py文件.This part is responsible训练模型

2.3 ACNet.py

ACNet.py中的_build_net()The function corresponds to the network structure in the paper：

2.4 mapf_gym环境

2.4.1 mapf_gym与mapf_gym_cap区别

继承了gym库,Used to build the environment
奇怪的是,它与mapf_gym_cap.pyThe code is almost the same,只是_observe()Functions are not the same：
mapf_gym中限制了mag（agent距离goal的位置）的大小,It must be the limited field of view：

githubThe explanation for these two documents in ：
mapf_gym：Multi-agent path planning gym environment, in which agents learn collective path planning
mapf_gym_cap.py：Multi-agent path planning gym environment, with capped goal distance state value for validation in larger environments

看了一下代码,It is called during trainingmapf_gym,测试的时候是mapf_gym_cap.py
Running these two codes directly will give an errorNameError: name 'coordinationRatio' is not defined,没查到coordinationRatio这个函数是干什么的,I put the corresponding codeprint(coordinationRatio(env))注释掉了.

2.4.2 搭建环境

A3C_RNN.py中创建环境：gym=mapf_gym.MAPFEnv(num_agents=n, world0=world[0],goals0=world[1])

2.5 mapgenerator.py

利用tk()库,生成环境,即手动设置obstacle,agent的位置

2.6 primal_testing.py

加载模型,进行测试,报错[Errno 2] No such file or directory: 'saved_environments/4_agents_10_size_0_density_id_0_environment.npy',I can't find where to generate this file.

2.7 unittest

报错：pyglet.canvas.xlib.NoSuchDisplayException: Cannot connect to "None",It is said on the Internet that you can run it directly in the terminal
But,run in terminal,也报错your graphic drives do not support OpenGL 2.0,网上查了一下,得有GPU才行.

① 试试这个方法：Image rendering in virtual formserver端启动虚拟化的图形渲染;没有用...
② 再试试这个pyglet.canvas.xlib.NoSuchDisplayException: Cannot connect to “None” ,It is also necessary to install half of it and find itGPU才行

Look at the code again,I found that the author commented it out at the beginningfrom gym.envs.classic_control import rendering 这个包,So I also commented out... But after commenting out,You can't see the graphical interface.

GroupLock.py

Responsible for multithreading：
Python多线程编程(一）：threading 模块 Thread Class usage details
Python 多线程编程(二):threading 模块中 Lock Class usage details
【python】详解threading模块：Condition类的使用（三）

3.报错合集：

1. 报错 Connection to Python debugger failed: Socket operation on nonsocket: configureBlocking
2. ImportError: DLL load failed: 找不到指定的模块. 解决办法：重新安装numpy和tensorflow
3. 运行mapgenerator.py代码时报错：TclError: no display name and no $DISPLAY environment variable,解决办法 Specify the display device manually： ① 终端输入printenv grep DISPLAY,查看版本,My output here islocalhost:10.0;② 再将root = Tk()改成root = Tk(screenName = ':10.0') ③ import的部分加上import matplotlib,matplotlib.use('Agg')