Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Divide-and-Conquer Reinforcement Learning

Dibya Ghosh, Avi Singh, Aravind Rajeswaran, Vikash Kumar, Sergey Levine

TL;DR

The paper tackles reinforcement learning in environments with highly diverse initial states, where standard policy-gradient methods struggle due to high gradient variance. It introduces divide-and-conquer RL (DnC), which partitions the initial state space into context slices, trains context-specific policies with KL-based coupling, and distills these into a single global policy. By learning in localized contexts and enforcing cross-context consistency, DnC achieves substantial performance gains on challenging robotic manipulation and locomotion tasks, outperforming TRPO, Distral, and ablated variants. The approach offers improved learning efficiency and robustness in complex, stochastic domains, with potential for more scalable context sharing in future work.

Abstract

Standard model-free deep reinforcement learning (RL) algorithms sample a new initial state for each trial, allowing them to optimize policies that can perform well even in highly stochastic environments. However, problems that exhibit considerable initial state variation typically produce high-variance gradient estimates for model-free RL, making direct policy or value function optimization challenging. In this paper, we develop a novel algorithm that instead partitions the initial state space into "slices", and optimizes an ensemble of policies, each on a different slice. The ensemble is gradually unified into a single policy that can succeed on the whole state space. This approach, which we term divide-and-conquer RL, is able to solve complex tasks where conventional deep RL methods are ineffective. Our results show that divide-and-conquer RL greatly outperforms conventional policy gradient methods on challenging grasping, manipulation, and locomotion tasks, and exceeds the performance of a variety of prior methods. Videos of policies learned by our algorithm can be viewed at http://bit.ly/dnc-rl

Divide-and-Conquer Reinforcement Learning

TL;DR

The paper tackles reinforcement learning in environments with highly diverse initial states, where standard policy-gradient methods struggle due to high gradient variance. It introduces divide-and-conquer RL (DnC), which partitions the initial state space into context slices, trains context-specific policies with KL-based coupling, and distills these into a single global policy. By learning in localized contexts and enforcing cross-context consistency, DnC achieves substantial performance gains on challenging robotic manipulation and locomotion tasks, outperforming TRPO, Distral, and ablated variants. The approach offers improved learning efficiency and robustness in complex, stochastic domains, with potential for more scalable context sharing in future work.

Abstract

Standard model-free deep reinforcement learning (RL) algorithms sample a new initial state for each trial, allowing them to optimize policies that can perform well even in highly stochastic environments. However, problems that exhibit considerable initial state variation typically produce high-variance gradient estimates for model-free RL, making direct policy or value function optimization challenging. In this paper, we develop a novel algorithm that instead partitions the initial state space into "slices", and optimizes an ensemble of policies, each on a different slice. The ensemble is gradually unified into a single policy that can succeed on the whole state space. This approach, which we term divide-and-conquer RL, is able to solve complex tasks where conventional deep RL methods are ineffective. Our results show that divide-and-conquer RL greatly outperforms conventional policy gradient methods on challenging grasping, manipulation, and locomotion tasks, and exceeds the performance of a variety of prior methods. Videos of policies learned by our algorithm can be viewed at http://bit.ly/dnc-rl

Paper Structure

This paper contains 26 sections, 12 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Divide-and-Conquer Reinforcement Learning
  5. Learning Policies for MDPs with Context
  6. The Divide-and-Conquer Reinforcement Learning Algorithm
  7. Experimental Evaluation
  8. Robotic Manipulation
  9. Picking.
  10. Lobbing.
  11. Catching.
  12. Locomotion
  13. Ant Position
  14. Stairs
  15. Discussion and Future Work
  16. ...and 11 more sections

Figures (2)

  • Figure 1: Average return and success rate learning curves of the global policy on Picking, Lobbing, Catching, Ant, and Stairs when partitioned into $4$ contexts. Metrics are evaluated each iteration on the global policy distilled from the current local policies at that iteration. On all of the tasks, DnC RL achieves the best results. On the Catching and Ant tasks, DnC performs comparably to the centralized variant, while on the Picking, Lobbing, and Stairs tasks, the full algorithm outperforms all others by a wide margin. All of the experiments are shown with 5 random seeds.
  • Figure :