Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Parallel Reinforcement Learning Simulation for Visual Quadrotor Navigation

Jack Saunders, Sajad Saeedi, Wenbin Li

TL;DR

A simulation framework, built on AirSim, which provides efficient parallel training and Ape-X is modified to incorporate parallel training of AirSim environments to make use of numerous networked computers.

Abstract

Reinforcement learning (RL) is an agent-based approach for teaching robots to navigate within the physical world. Gathering data for RL is known to be a laborious task, and real-world experiments can be risky. Simulators facilitate the collection of training data in a quicker and more cost-effective manner. However, RL frequently requires a significant number of simulation steps for an agent to become skilful at simple tasks. This is a prevalent issue within the field of RL-based visual quadrotor navigation where state dimensions are typically very large and dynamic models are complex. Furthermore, rendering images and obtaining physical properties of the agent can be computationally expensive. To solve this, we present a simulation framework, built on AirSim, which provides efficient parallel training. Building on this framework, Ape-X is modified to incorporate decentralised training of AirSim environments to make use of numerous networked computers. Through experiments we were able to achieve a reduction in training time from 3.9 hours to 11 minutes using the aforementioned framework and a total of 74 agents and two networked computers. Further details including a github repo and videos about our project, PRL4AirSim, can be found at https://sites.google.com/view/prl4airsim/home

Parallel Reinforcement Learning Simulation for Visual Quadrotor Navigation

TL;DR

A simulation framework, built on AirSim, which provides efficient parallel training and Ape-X is modified to incorporate parallel training of AirSim environments to make use of numerous networked computers.

Abstract

Reinforcement learning (RL) is an agent-based approach for teaching robots to navigate within the physical world. Gathering data for RL is known to be a laborious task, and real-world experiments can be risky. Simulators facilitate the collection of training data in a quicker and more cost-effective manner. However, RL frequently requires a significant number of simulation steps for an agent to become skilful at simple tasks. This is a prevalent issue within the field of RL-based visual quadrotor navigation where state dimensions are typically very large and dynamic models are complex. Furthermore, rendering images and obtaining physical properties of the agent can be computationally expensive. To solve this, we present a simulation framework, built on AirSim, which provides efficient parallel training. Building on this framework, Ape-X is modified to incorporate decentralised training of AirSim environments to make use of numerous networked computers. Through experiments we were able to achieve a reduction in training time from 3.9 hours to 11 minutes using the aforementioned framework and a total of 74 agents and two networked computers. Further details including a github repo and videos about our project, PRL4AirSim, can be found at https://sites.google.com/view/prl4airsim/home
Paper Structure (11 sections, 2 figures)

This paper contains 11 sections, 2 figures.

Table of Contents

  1. Introduction
  2. Literature
  3. Reinforcement Learning
  4. Improving Data Efficiency
  5. Distributed Reinforcement Learning Frameworks
  6. Examples of using Distributed Frameworks
  7. Method
  8. Parallel image rendering
  9. Parallel Reinforcement Learning architecture
  10. Result
  11. Conclusion

Figures (2)

  • Figure 1: Average time to collect all images for a set number of drones within the simulator. Each data point was the average of 1000 function calls.
  • Figure 2: