Scalable Strategies for Continual Learning with Replay

TL;DR

This paper tackles the scalability challenge of replay-based continual learning by integrating three complementary strategies: low-rank adaptation (LoRA) for parameter-efficient, regularized updates; a phasic consolidation phase to dramatically improve replay efficiency; and model merging (including sequential merging and EMA) to compose knowledge across tasks. It analytically compares LoRA to full fine-tuning, introduces consolidation with definitions of CSR and TRP, and adapts model merging to the continual setting, all while allowing replay to function synergistically with these methods. Key findings include up to 65% fewer replay samples needed to achieve baseline performance when combining strategies, and competitive accuracy relative to online EMA through sequential merging. The work demonstrates a scalable toolset for replay-enabled continual learning, with strong implications for deployment in dynamic, open-ended environments and potential applicability to reinforcement learning contexts.

Abstract

Future deep learning models will be distinguished by systems that perpetually learn through interaction, imagination, and cooperation, blurring the line between training and inference. This makes continual learning a critical challenge, as methods that efficiently maximize bidirectional transfer across learning trajectories will be essential. Replay is on track to play a foundational role in continual learning, allowing models to directly reconcile new information with past knowledge. In practice, however, replay is quite unscalable, doubling the cost of continual learning when applied naively. Moreover, the continual learning literature has not fully synchronized with the multi-task fine-tuning literature, having not fully integrated highly scalable techniques like model merging and low rank adaptation into a replay-enabled toolset that can produce a unified model in the face of many sequential tasks. In this paper, we begin by applying and analyzing low rank adaptation in a continual learning setting. Next, we introduce consolidation, a phasic approach to replay which leads to up to 55\% less replay samples being needed for a given performance target. Then, we propose sequential merging, an offshoot of task arithmetic which is tailored to the continual learning setting and is shown to work well in combination with replay. Finally, we demonstrate that the developed strategies can operate synergistically, resulting in a highly scalable toolset that outperforms standalone variants.

Figures (6)

• Figure 1: Visual representation of the impact of changing the replay ratio (RR). Each block represents the samples used during learning of a single task. Reducing the replay ratio allows us to allocate more of the training batch to task samples, which reduces the total number of samples processed and, therefore, reduces training time.
• Figure 2: LoRA performance vs. FFT performance across task sizes while keeping total downstream classes fixed at 120 (top). LoRA performance vs. FFT performance across replay ratios (bottom). Note that "Continual Pre-Training (PT)" indicates pre-training accuracy at the end of training while "Continual Pre-Training (1:N)" indicates average downstream task accuracy at the end of training.
• Figure 3: Visual representation of continual learning with consolidation. Standard continual learning complements task samples with many replay samples per batch, and does not have a post-task training phase. With consolidation, we reduce the replay ratio during task learning (optionally using LoRA depending on the training distribution) then execute a post-task training phase on a distribution optimized for reconciliation of old and new knowledge. This allows a vastly reduced number of total gradient steps across training.
• Figure 4: Performance of standard replay (no consolidation) versus consolidation (CIL/CPT). Left: mean final accuracy across tasks for each replay-ratio (RR) and consolidation-sample-ratio (CSR) setting, with TRP reporting the total proportion of replayed samples. Note that a CSR of 0.0 indicates standard replay, which acts as a baseline. Right: violin plots of per-task accuracies at the end of training.
• Figure 5: Visual representation of the differences between parallel and sequential merging. Parallel merging trains $N$ independent, task-specific models based on the same pre-trained backbone, then merges their weights to produce a model that can perform tasks $1-N$. Sequential merging trains on each task sequentially, merging the pre- and post-task model checkpoint after training on each task.