Task-Core Memory Management and Consolidation for Long-term Continual Learning

TL;DR

This work tackles long-term continual learning by formalizing a realistic long-horizon setup and proposing Long-CL, a memory-inspired framework with Task-Core Memory Management (MemMan) and Long-term Memory Consolidation (MemCon). It introduces two benchmarks, MMLongCL-Bench (multimodal) and TextLongCL-Bench (textual), to assess robustness across diverse tasks and distributions. Empirical results show that Long-CL achieves state-of-the-art performance, significantly reducing forgetting while approaching multitask performance, with Ablation analyses confirming the effectiveness of both MemMan and MemCon. The proposed benchmarks and methods offer a practical pathway for scalable, memory-aware continual learning in large models, with future directions including cross-modal task integration.

Abstract

In this paper, we focus on a long-term continual learning (CL) task, where a model learns sequentially from a stream of vast tasks over time, acquiring new knowledge while retaining previously learned information in a manner akin to human learning. Unlike traditional CL settings, long-term CL involves handling a significantly larger number of tasks, which exacerbates the issue of catastrophic forgetting. Our work seeks to address two critical questions: 1) How do existing CL methods perform in the context of long-term CL? and 2) How can we mitigate the catastrophic forgetting that arises from prolonged sequential updates? To tackle these challenges, we propose a novel framework inspired by human memory mechanisms for long-term continual learning (Long-CL). Specifically, we introduce a task-core memory management strategy to efficiently index crucial memories and adaptively update them as learning progresses. Additionally, we develop a long-term memory consolidation mechanism that selectively retains hard and discriminative samples, ensuring robust knowledge retention. To facilitate research in this area, we construct and release two multi-modal and textual benchmarks, MMLongCL-Bench and TextLongCL-Bench, providing a valuable resource for evaluating long-term CL approaches. Experimental results show that Long-CL outperforms the previous state-of-the-art by 7.4\% and 6.5\% AP on the two benchmarks, respectively, demonstrating the effectiveness of our approach.

Figures (4)

• Figure 1: The performance of our method and O-LoRA on MMLongCL-Bench.
• Figure 2: The performance heatmap of our method and O-LoRA on MMLongCL-Bench. The horizontal axis indicates the training order and the vertical axis represents the performance after tuning.
• Figure 3: The framework of our Long-CL. First, we introduce MemMan to index the task-core memory and adjust the model's memory adaptively based on the relationship between the current and previous tasks. Then, MemCon learns the key knowledge using hard sample selection and differential sample selection to choose task-relevant informative (i.e., the samples in the green annulus) and cross-task generalizable samples (i.e., the samples in the yellow annulus), respectively.
• Figure 4: Performance(%) of Long-CL with different buffer size and $K$ value on MMLongCL-Bench.