Continual Knowledge Adaptation for Reinforcement Learning

TL;DR

CKA-RL tackles continual reinforcement learning by introducing a task-specific knowledge vector pool that enables dynamic reuse of historical knowledge while mitigating forgetting. It comprises three components: Knowledge Vectors to encode task adaptations, Continual Knowledge Adaptation to fuse past vectors with a new task via adaptive coefficients, and Adaptive Knowledge Merging to bound memory by merging similar vectors. The approach yields state-of-the-art improvements in average performance and forward transfer across Meta-World, SpaceInvaders, and Freeway, while maintaining a bounded memory footprint and fast inference. This work provides a scalable, transfer-friendly mechanism for continual learning in non-stationary environments and demonstrates clear practical benefits for multi-task RL systems.

Abstract

Reinforcement Learning enables agents to learn optimal behaviors through interactions with environments. However, real-world environments are typically non-stationary, requiring agents to continuously adapt to new tasks and changing conditions. Although Continual Reinforcement Learning facilitates learning across multiple tasks, existing methods often suffer from catastrophic forgetting and inefficient knowledge utilization. To address these challenges, we propose Continual Knowledge Adaptation for Reinforcement Learning (CKA-RL), which enables the accumulation and effective utilization of historical knowledge. Specifically, we introduce a Continual Knowledge Adaptation strategy, which involves maintaining a task-specific knowledge vector pool and dynamically using historical knowledge to adapt the agent to new tasks. This process mitigates catastrophic forgetting and enables efficient knowledge transfer across tasks by preserving and adapting critical model parameters. Additionally, we propose an Adaptive Knowledge Merging mechanism that combines similar knowledge vectors to address scalability challenges, reducing memory requirements while ensuring the retention of essential knowledge. Experiments on three benchmarks demonstrate that the proposed CKA-RL outperforms state-of-the-art methods, achieving an improvement of 4.20% in overall performance and 8.02% in forward transfer. The source code is available at https://github.com/Fhujinwu/CKA-RL.

Figures (11)

• Figure 1: An illustration of the CKA-RL. When learning a new task $\tau_{k}$, the agent $\pi_k$ adapts by using historical knowledge vectors $\mathcal{V} = \{\bm{v}_1, \ldots, \bm{v}_{k-1}\}$ with $|\mathcal{V}|\leq K_{\text{max}}$, and learning a new task-specific knowledge vector $\bm {v}_{k}$, while the base parameter $\bm{\theta}_{\text{base}}$ remains fixed. After training, the new knowledge vector $\bm{v}_{k}$ is added to a knowledge vector pool $\mathcal{V}$. To maintain memory efficiency, we merge the most similar pairs of knowledge vectors $(\bm{v}_m,\bm{v}_n)$ in $\mathcal{V}$ into $\bm{v}_{\text{merge}}$ when $|\mathcal{V}|>K_{\text{max}}$, thus ensuring essential knowledge is retained while supporting scalable continual learning across sequential tasks.
• Figure 2: Comparison of SOTA methods with the proposed CKA-RL in terms of average forward transfer and average performance.
• Figure 3: Effects of different pool size $K_{\text{max}}$ in Adaptive Knowledge Merging in the Freeway sequence. Our proposed CKA-RL achieves excellent performance when $K_{\text{max}}=5$.
• Figure 4: Reward curve comparison in the SpaceInvaders sequence. CKA-RL attains a higher initial reward and converges faster, underscoring superior use of historical vectors.
• Figure 5: Analysis of Total Parameter Memory Consumption and Inference Latency: (a) demonstrates that CKA-RL maintains nearly constant parameter and activation size through vector merging, while others grow linearly; (b) shows that CKA-RL maintains nearly constant inference latency, while others suffer from increasing computational overhead. More details can be found in Appendix \ref{['appendix:efficiency']}.