Continual Offline Reinforcement Learning via Diffusion-based Dual Generative Replay

TL;DR

Continual offline RL (CORL) seeks to learn a sequence of tasks from static offline data while avoiding forgetting and enabling forward transfer. The authors propose CuGRO, a diffusion-based dual generative replay framework that decouples policy representation into a diffusion-based behavior model $\mu_{\bm{\phi}}(\bm{a}|\bm{s})$ and a task-conditioned state generator $p_{\bm{\varphi}}(\bm{s}|k)$, enabling high-fidelity replay of past tasks without storing real samples. A diffusion-based dual generator system interleaves replayed pseudo samples with real data and uses a multi-head critic with behavior cloning to mitigate forgetting. Experiments on MuJoCo and Meta-World show improved forward transfer and reduced forgetting, closely matching performance obtained with ground-truth past data; the authors release code at the provided URL.

Abstract

We study continual offline reinforcement learning, a practical paradigm that facilitates forward transfer and mitigates catastrophic forgetting to tackle sequential offline tasks. We propose a dual generative replay framework that retains previous knowledge by concurrent replay of generated pseudo-data. First, we decouple the continual learning policy into a diffusion-based generative behavior model and a multi-head action evaluation model, allowing the policy to inherit distributional expressivity for encompassing a progressive range of diverse behaviors. Second, we train a task-conditioned diffusion model to mimic state distributions of past tasks. Generated states are paired with corresponding responses from the behavior generator to represent old tasks with high-fidelity replayed samples. Finally, by interleaving pseudo samples with real ones of the new task, we continually update the state and behavior generators to model progressively diverse behaviors, and regularize the multi-head critic via behavior cloning to mitigate forgetting. Experiments demonstrate that our method achieves better forward transfer with less forgetting, and closely approximates the results of using previous ground-truth data due to its high-fidelity replay of the sample space. Our code is available at \href{https://github.com/NJU-RL/CuGRO}{https://github.com/NJU-RL/CuGRO}.

Figures (11)

• Figure 1: Sequential training of CuGRO. (a) A new diffusion-based state generative model is trained to mimic a mixed data distribution of real samples $\bm{s}$ and replayed ones $\hat{\bm{s}}$. (b) A new diffusion-based behavior generative model learns from real state-action pairs $(\bm{s},\bm{a})$ and pseudo pairs $(\hat{\bm{s}}, \hat{\bm{a}})$, where replayed action $\hat{\bm{a}}$ is obtained by feeding replayed states $\hat{\bm{s}}$ into current behavior generator. (c) A new head in the critic is expanded for tackling the new task with real state-action pairs and Bellman targets, and previous heads are regularized by cloning the Q-value of replayed pairs $(\hat{\bm{s}}, \hat{\bm{a}})$.
• Figure 2: Performance of CuGRO and baselines on cumulative tasks during sequential training evaluated on MuJoCo and Meta-World. The policy is trained with $100$ iterations for each task, and each iteration involves feeding all data batches to the policy model for training.
• Figure 3: Performance of CuGRO with different replay variants over cumulative tasks during sequential training.
• Figure 4: Performance of CuGRO with different generative models over cumulative tasks during sequential training.
• Figure 5: Performance of CuGRO with varying coefficients $\lambda$ over cumulative tasks during sequential training.