Continual Task Learning through Adaptive Policy Self-Composition

TL;DR

This study develops the Offline Continual World benchmark and demonstrates that traditional CL methods struggle with catastrophic forgetting, primarily due to the unique distribution shifts inherent to CORL scenarios, and introduces CompoFormer, a structure-based continual transformer model that adaptively composes previous policies via a meta-policy network.

Abstract

Training a generalizable agent to continually learn a sequence of tasks from offline trajectories is a natural requirement for long-lived agents, yet remains a significant challenge for current offline reinforcement learning (RL) algorithms. Specifically, an agent must be able to rapidly adapt to new tasks using newly collected trajectories (plasticity), while retaining knowledge from previously learned tasks (stability). However, systematic analyses of this setting are scarce, and it remains unclear whether conventional continual learning (CL) methods are effective in continual offline RL (CORL) scenarios. In this study, we develop the Offline Continual World benchmark and demonstrate that traditional CL methods struggle with catastrophic forgetting, primarily due to the unique distribution shifts inherent to CORL scenarios. To address this challenge, we introduce CompoFormer, a structure-based continual transformer model that adaptively composes previous policies via a meta-policy network. Upon encountering a new task, CompoFormer leverages semantic correlations to selectively integrate relevant prior policies alongside newly trained parameters, thereby enhancing knowledge sharing and accelerating the learning process. Our experiments reveal that CompoFormer outperforms conventional CL methods, particularly in longer task sequences, showcasing a promising balance between plasticity and stability.

Figures (8)

• Figure 1: Adaptive policy self-composition architecture. When a new task arises (represented by a star), its textual description is processed by the frozen S-BERT model to compute attention scores with previous task descriptions. After several update iterations of the attention module, if the composed policy is enough for the current task (the performance exceeds a predefined threshold), it is used directly; otherwise, new parameters are incorporated alongside the composed policy to construct the new policy $\pi^{(k)}$.
• Figure 2: The architecture of the meta-policy network. The core module is built upon the Transformer architecture, which receives the trajectory as input and outputs the corresponding action. Upon encountering task ${\mathcal{M}}^{(k)}$, our method presents two variants: the "Grow" variant, which adds new parameters to the Transformer in a LoRA format, and the "Prune" variant, which utilizes a masking technique to deactivate certain parameters within the Transformer (where green indicates activated parameters and grey denotes inactivated ones).
• Figure 3: Performance across 3 random seeds for all methods on the OCW20 sequence. CompoFormer-Grow and Prune outperform all baselines, demonstrating faster task adaptation.
• Figure 4: (a) Visualization of attention scores from the self-composing policy module in the OCW10 benchmark with the CompoFormer-Grow edition, where the diagonal is excluded and set to 1. (b) Evaluation of the effectiveness of our self-composing policy module through three variants: Sharing, Addition, and Selection, conducted in the OCW10 benchmark for both the CompoFormer-Grow and CompoFormer-Prune editions. Each result is averaged over three random seeds.
• Figure 5: Performance across three random seeds for each task in the OCW10 benchmark. "Single" refers to the performance of individual task training, while the other methods reflect each task's performance after the entire learning process.