Reinforcement Learning - norlab-ulaval/SNOW-sim_internship_H22 GitHub Wiki

SSIM-Sim (Youtrack)  •  SNOW (WebSite)  •  isabelleysseric (Docker Hub)  •  SNOW-sim (GitHub)  •  Wiki SNOW-sim (Wiki)

Hello, during my research on simulators I found some articles and books that helped me in my work. I share my sources with you in the hope that it helps you too. If you have other interesting resources to share on the subject, do not hesitate.

And below you will find a section for troubleshooting with the installation of Gym-Ignition.

Figure: PacMan game and Reinforcement Learning problem

*RL: Reinforcement Learning

• Learning to walk in minutes using massively parallel Deep Reinforcement learning (30 Oct 2021) by Nikita Rudin, David Hoeller, Philipp Reist, Marco Hutter
• Sim-to-Real transfer in Deep Reinforcement Learning for robotics: a survey (8 Jul 2021) by Wenshuai Zhao, Jorge Peña Queralta, Tomi Westerlund
• Benchmarking Deep Reinforcement Learning for continous control (27 May 2016) by Yan Duan, Xi Chen, Rein Houthooft, John Schulman, Pieter Abbeel
• Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning (25 Aug 2021) by Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, Gavriel State
• High-Fidelity Yet Fast Dynamic Models of Wheeled Mobile Robots (June 2016) by Neal Seegmiller, Alonzo Kelly
• End-to-End Reinforcement Learning for Multi-agent continous control (Dec. 2019) by Zilong Jiao, Jae Oh
• Sim2Real learning of obstacle avoidance for robotic manipulators in uncertain environments (30 September 2021) by Kefang Zhang, Jiatao Lin, Lv Bi, Tan Zhang
• Reinforcement Learning Control of a Forestry Crane Manipulator (4 March 2021) by Jennifer Andersson, Kenneth Bodin, Daniel Lindmark, Martin Servin, Erik Wallin
• Comparing Popular Simulation Environments in the Scope of Robotics and Reinforcement Learning (8 March 2021) by Marian Körber, Johann Lange, Stephan Rediske, Simon Steinmann, Roland Glück
• OpenAI Gym (5 June 2016) by Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, Wojciech Zaremba
• Gym-Ignition reproducible robotic simulations for Reinforcement Learning (5 June 2016) by Diego Ferigo, Silvio Traversaro, Giorgio Metta, Daniele Pucci
• Multi-task Learning for Continuous Control (3 Feb 2018) by Himani Arora, Rajath Kumar, Jason Krone, Chong Li ...

*AV: Autonomous Vehicles
*SIM: Simulators

• Kilometer-scale autonomous navigation in subarctic forests: challenges and lessons learned (27 Nov 2021) by Dominic Baril, Simon-Pierre Deschênes, Olivier Gamache, Maxime Vaidis, Damien LaRocque, Johann Laconte, Vladimír Kubelka, Philippe Giguère, François Pomerleau
• A review of physics simulators for robotic applications (25 March 2021) by Jack Collins, Shelvin Chand, Anthony Vanderkop, David Howard
• CARLA an open urban driving simulator
• Simulator AirSim - High fidelity visual and physical simulation for autonomous vehicles
  ...

• Fundamentals of Applied Dynamics (Springer, 2004) by Roberto A. Tenenbaum
• Physics-based virtual environments for autonomous earthmoving and mining machinery (January 2018) by Martin Servin, Michael Brandl
• Springer Handbook of Robotics (Springer, 2016) by Bruno Siciliano, Oussama Khatib ...

Figure: OpenAI environment

OpenAI Gym is a toolkit for developing and compar ing reinforcement learning algorithms. Here, a list of environments based on OpenAi Gym for vehicles.

• Gym-Ignition is designed for repeatable results and allows parallel or headless mode simulation while providing commo n rigid body dynami cs utilities.
• Wheeled Mobile Robot Dynamics Engine (WMRDE) is for simulation and model ing of wheeled mobi le robots (WMR) and t racked vehicles.
• Highway-Env : A collection of envi ronments for autonomous driving and tactical decision-making tasks.
• Gym Electric Motor (GEM) is an OpenAI Gym environment for simulating electric motor control and RL experiments.
• MuJoCo (Multi-Joint dynamics with Contact) is intended for cont inuous control tasks, performed in a multi-body dynami cs simulator with contact.
• Gym-Chrono is a set of cont inuous state and action space DRL environm ents based on the C hrono physics eng ine.

Web Sites:

OpenAI Gym ScenarIO and gym-ignition Gym-Ignition environment
Stable Baselines
OpenAI Spinning Up

Courses: Markov Decision Process (MDP) and Dynamic Programming

CS 188 | Introduction to Artificial Intelligence (Fall 2018, Week 4)

• RL | Video 1 - Slides
• RL | Video 2 - Slides

Courses: Reinforcement Learning

CS 188 | Introduction to Artificial Intelligence (Fall 2018, Week 5)

• RL | Video 1 - Slides
• RL | Video 2 - Slides

Courses: Deep Reinforcement Learning:

CS 285 | Deep Reinforcement Learning
OpenAI Spinning Up
Stable Baselines OpenAI Gym

Courses: Advanced Topics in Learning and Decision Making:

CS 294-190 | Advanced topics in learning and decision making (Fall 2021)

• DRL | Videos
• RL and Language (Week 10)
  Learning to Win by Reading Manuals in a Monte-Carlo Framework (30 April 2012) by S.R.K. Branavan, David Silver, Regina Barzilay

Courses: Advanced Robotics:

CS 287: Advanced Robotics by Pieter Abbeel, Fall 2019

• Videos and Website

Books:

Reinforcement Learning an Introduction (Bradford Books, 2018) by Richard S. Sutton and Barto

Figure: troubleshooting-it-admin from Tenor site

During my project on reinforcement learning for the Gazebo Ignition simulator (Fortress version) I encountered some problems during the installation of Gym-Ignition and I found solutions. Some solutions solved my problems, some not completely, so if you have better solutions or other problems you solved that you would like to share, feel free to share them, I will add them to the list.

Thank you for your contribution and hoping that this section can help you too.

- PROBLEM 1 : ERROR: Could not find a version that satisfies the requirement …    

Problem:

ERROR: Could not find a version that satisfies the requirement gy_ignition_models (from versions: none)
ERROR: No matching distribution found for gy_ignition_models

Solution:

pip install gym-ignition-models

- PROBLEM 2 : WARNING: You are using pip version … ; however, version … is available.   

Problem:

WARNING: You are using pip version 21.3.1; however, version 22.0.4 is available.
You should consider upgrading via the '/home/norlab/PycharmProjects/Ignition/venv/bin/python -m pip install --upgrade pip' command.

Solution:

/home/norlab/PycharmProjects/Ignition/venv/bin/python -m pip install --upgrade pip  

- PROBLEM 3 : gym-ignition-models   

Solution:

# From PyPI (release versions)
pip3 install gym-ignition-models

# From PyPI (pre-release versions)
pip3 install --pre gym-ignition-models

# From the repository
pip3 install git+https://github.com/robotology/gym-ignition-models.git

- PROBLEM 4 : E: Unable to locate package colcon    

Problem:

sudo apt-get install colcon
E: Unable to locate package colcon

Solution:

sudo apt install python3-colcon-common-extensions  

- PROBLEM 5 : ImportError: cannot import name 'gazebo' from partially initialized module 'scenario'      

Problem:

from scenario import gazebo as scenario_gazebo
ImportError: cannot import name 'gazebo' from partially initialized module 'scenario' (most likely due to a circular import) (/home/norlab/PycharmProjects/Ignition/scenario.py)

Solution: I don't have the solution yet

- PROBLEM 6 : QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime'    

Link

Problem:

QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime'

Solution:

export XDG_RUNTIME_DIR=/some/directory/you/specify 
export RUNLEVEL=3
source .bashrc