Collective Robot Reinforcement Learning with Distributed Asynchronous Guided Policy Search

TL;DR

The paper addresses the challenge of learning generalizable visuomotor policies for real-world robotics under diverse conditions.It introduces asynchronous distributed guided policy search (ADGPS), which parallelizes data collection across multiple robots and trains a single global neural policy with a replay buffer.Local policies are learned per-robot using trajectory-centric RL (LQR or PI^2), while the global policy is learned via supervised learning from aggregated experiences.Experiments in simulation and on a four-robot door-opening task show faster training times, better utilization, and improved generalization compared to single-robot GPS.

Abstract

In principle, reinforcement learning and policy search methods can enable robots to learn highly complex and general skills that may allow them to function amid the complexity and diversity of the real world. However, training a policy that generalizes well across a wide range of real-world conditions requires far greater quantity and diversity of experience than is practical to collect with a single robot. Fortunately, it is possible for multiple robots to share their experience with one another, and thereby, learn a policy collectively. In this work, we explore distributed and asynchronous policy learning as a means to achieve generalization and improved training times on challenging, real-world manipulation tasks. We propose a distributed and asynchronous version of Guided Policy Search and use it to demonstrate collective policy learning on a vision-based door opening task using four robots. We show that it achieves better generalization, utilization, and training times than the single robot alternative.

Figures (9)

• Figure 1: Multiple robots collaborating to learn the door opening skill. Our system allows the robots to operate continuously to collect a large amount of diverse experience, while the policy is simultaneously trained with a replay buffer of the latest trajectory samples.
• Figure 2: Diagram of the training loop for synchronous GPS with a single replica. Rollout execution corresponds to line 2 in Algorithm \ref{['alg:gps']}, local policy optimization to line 3, and global policy optimization to line 4. In BADMM-based GPS, the algorithm additionally alternates between local and global policy optimization multiple times before executing new rollouts. This sequential version of the algorithm requires training to pause while performing rollouts, and vice versa.
• Figure 3: The training loop for ADGPS with multiple replicas. Rollout execution and global policy optimization are decoupled via the replay memory. Multiple robots concurrently collect data and asynchronously update the parameter server, allowing maximal utilization of both computational and robot resources, as well as parallelization across multiple robots and servers.
• Figure 4: Average costs of the 4 test instances used in the simulated reaching task, over number of iterations (left) as well as training duration (right). ADGPS-4 and ADGPS-8 denote 4 and 8 pairs of local and global workers, respectively, while AGPS is an asynchronous run with a single pair of workers. Note that asynchronous training does slightly reduce the improvement per iteration, but substantially improves training time when multiple workers are used.
• Figure 5: Speedup in wall-clock training time and sample count comparison between GPS and the asynchronous variants, measured as the wall-clock time or sample count needed to reach a threshold cost value. Note that additional asynchronous workers incur only a modest cost in total sample count, while providing a substantial improvement in wall-clock training time.