AdaER: An Adaptive Experience Replay Approach for Continual Lifelong Learning

TL;DR

AdaER tackles catastrophic forgetting in class-IL continual learning by integrating two innovations: Contextually-Cued Memory Recall (C-CMR) for selective replay based on data- and task-conflicts, and Entropy-Balanced Reservoir Sampling (E-BRS) for balanced, information-rich memory updates. The memory buffer stores samples with associated task IDs, enabling memory-aware rehearsal and transfers across tasks. Empirical results across Split-MNIST, Split-FMNIST, Split-CIFAR10, and Split-CIFAR100 show AdaER outperforms strong ER-based baselines, with improved accuracy, reduced forgetting, and favorable backward transfer, particularly under memory constraints and imbalanced data. The work advances practical continual learning by addressing both replay selection and buffer diversification, with implications for scalable lifelong learning systems.

Abstract

Continual lifelong learning is an machine learning framework inspired by human learning, where learners are trained to continuously acquire new knowledge in a sequential manner. However, the non-stationary nature of streaming training data poses a significant challenge known as catastrophic forgetting, which refers to the rapid forgetting of previously learned knowledge when new tasks are introduced. While some approaches, such as experience replay (ER), have been proposed to mitigate this issue, their performance remains limited, particularly in the class-incremental scenario which is considered natural and highly challenging. In this paper, we present a novel algorithm, called adaptive-experience replay (AdaER), to address the challenge of continual lifelong learning. AdaER consists of two stages: memory replay and memory update. In the memory replay stage, AdaER introduces a contextually-cued memory recall (C-CMR) strategy, which selectively replays memories that are most conflicting with the current input data in terms of both data and task. Additionally, AdaER incorporates an entropy-balanced reservoir sampling (E-BRS) strategy to enhance the performance of the memory buffer by maximizing information entropy. To evaluate the effectiveness of AdaER, we conduct experiments on established supervised continual lifelong learning benchmarks, specifically focusing on class-incremental learning scenarios. The results demonstrate that AdaER outperforms existing continual lifelong learning baselines, highlighting its efficacy in mitigating catastrophic forgetting and improving learning performance.

Figures (12)

• Figure 1: Split-MNIST: continual lifelong learning example.
• Figure 2: System diagram of the proposed AdaER: an adaptive experience replay algorithm with the developed contextually-cued memory recall (C-CMR) method for the replay stage and the entropy-balanced reservoir sampling (E-BRS) strategy for the update stage.
• Figure 3: Illustration of the developed C-CMR method: the most contextually-cued memories $\mathcal{R}$ are replayed to mitigate the forgetting with the combination of example-interfered buffer $\mathcal{R}_e$ and task-associated buffer $\mathcal{R}_t$.
• Figure 4: Task-associated interference-transfer relationship with a three-task continual learning scenario.
• Figure 5: The impact of different memory size over averaged testing accuracy.