Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Hierarchical Orchestra of Policies

Thomas P Cannon, Özgür Simsek

TL;DR

This paper introduces a modularity-based approach, called Hierarchical Orchestra of Policies (HOP), designed to mitigate catastrophic forgetting in lifelong reinforcement learning, which significantly outperforms baseline methods in retaining knowledge across tasks and performs comparably to state-of-the-art transfer methods that require task labelling.

Abstract

Continual reinforcement learning poses a major challenge due to the tendency of agents to experience catastrophic forgetting when learning sequential tasks. In this paper, we introduce a modularity-based approach, called Hierarchical Orchestra of Policies (HOP), designed to mitigate catastrophic forgetting in lifelong reinforcement learning. HOP dynamically forms a hierarchy of policies based on a similarity metric between the current observations and previously encountered observations in successful tasks. Unlike other state-of-the-art methods, HOP does not require task labelling, allowing for robust adaptation in environments where boundaries between tasks are ambiguous. Our experiments, conducted across multiple tasks in a procedurally generated suite of environments, demonstrate that HOP significantly outperforms baseline methods in retaining knowledge across tasks and performs comparably to state-of-the-art transfer methods that require task labelling. Moreover, HOP achieves this without compromising performance when tasks remain constant, highlighting its versatility.

Hierarchical Orchestra of Policies

TL;DR

This paper introduces a modularity-based approach, called Hierarchical Orchestra of Policies (HOP), designed to mitigate catastrophic forgetting in lifelong reinforcement learning, which significantly outperforms baseline methods in retaining knowledge across tasks and performs comparably to state-of-the-art transfer methods that require task labelling.

Abstract

Continual reinforcement learning poses a major challenge due to the tendency of agents to experience catastrophic forgetting when learning sequential tasks. In this paper, we introduce a modularity-based approach, called Hierarchical Orchestra of Policies (HOP), designed to mitigate catastrophic forgetting in lifelong reinforcement learning. HOP dynamically forms a hierarchy of policies based on a similarity metric between the current observations and previously encountered observations in successful tasks. Unlike other state-of-the-art methods, HOP does not require task labelling, allowing for robust adaptation in environments where boundaries between tasks are ambiguous. Our experiments, conducted across multiple tasks in a procedurally generated suite of environments, demonstrate that HOP significantly outperforms baseline methods in retaining knowledge across tasks and performs comparably to state-of-the-art transfer methods that require task labelling. Moreover, HOP achieves this without compromising performance when tasks remain constant, highlighting its versatility.

Paper Structure

This paper contains 9 sections, 3 equations, 5 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Hierarchical Orchestra of Policies
  3. Results
  4. Summary and Discussion
  5. Appendix
  6. HOP Algorithm details
  7. Experiment details
  8. PNN with PPO Algorithm Details
  9. Hierarchical Weighting Examples

Figures (5)

  • Figure 1: The flow of information as a HOP agent acts in a task.
  • Figure 2: Hierarchical formation of the fourth level of a HOP action policy.
  • Figure 3: Training performance of HOP, PNN and PPO on three experiments where environments are periodically changed. The red dashed lines indicate the points when the environment are switched. The green dashed lines show when HOP returns to the highest average evaluation reward achieved in the first environment before the change. The black dashed lines represents this point for PPO. Shaded areas are the standard error. All experiments are conducted with the Procgen easy setting.
  • Figure 4: From left to right, Climber, CoinRun, StarPilot and, Ninja. In our experiments the backgrounds are all black (use_backgrounds=False).
  • Figure 5: Separate Actor and Critic Networks for the ProcGen Architecture