Provable Effects of Data Replay in Continual Learning: A Feature Learning Perspective

TL;DR

This work analyzes full data replay in continual learning through a feature-learning lens under a multi-view data model. It shows that forgetting can persist when cumulative noise outweighs earlier signals, but sufficient accumulated signal enables recovery of earlier tasks, with an important role for task ordering. The authors derive SNR-based conditions that delineate regimes of persistent forgetting, enhanced learning, and eventual correct classification, and demonstrate both synthetic and CIFAR-100 experiments that corroborate the theory. The key practical implication is that designing order-aware replay strategies—prioritizing high-signal tasks—can improve retention and facilitate knowledge transfer in continual learning systems.

Abstract

Continual learning (CL) aims to train models on a sequence of tasks while retaining performance on previously learned ones. A core challenge in this setting is catastrophic forgetting, where new learning interferes with past knowledge. Among various mitigation strategies, data-replay methods, where past samples are periodically revisited, are considered simple yet effective, especially when memory constraints are relaxed. However, the theoretical effectiveness of full data replay, where all past data is accessible during training, remains largely unexplored. In this paper, we present a comprehensive theoretical framework for analyzing full data-replay training in continual learning from a feature learning perspective. Adopting a multi-view data model, we identify the signal-to-noise ratio (SNR) as a critical factor affecting forgetting. Focusing on task-incremental binary classification across $M$ tasks, our analysis verifies two key conclusions: (1) forgetting can still occur under full replay when the cumulative noise from later tasks dominates the signal from earlier ones; and (2) with sufficient signal accumulation, data replay can recover earlier tasks-even if their initial learning was poor. Notably, we uncover a novel insight into task ordering: prioritizing higher-signal tasks not only facilitates learning of lower-signal tasks but also helps prevent catastrophic forgetting. We validate our theoretical findings through synthetic and real-world experiments that visualize the interplay between signal learning and noise memorization across varying SNRs and task correlation regimes.

Figures (6)

• Figure 1: Illustration of feature signals across multiple tasks using images from Salient ImageNet.
• Figure 2: Dynamics of signal learning and noise memorization during full data-replay continual training across different task orderings and correlation strengths.
• Figure 3: Dynamics of signal learning and noise memorization under lower SNR.
• Figure 4: Accuracy under full data-replay continual training across different task orderings and correlation strengths.
• Figure 5: Catastrophic forgetting under full data-replay continual training across various correlation strengths.

Theorems & Definitions (39)

• Definition 1: Data Distribution for Task $m$
• Definition 2: Catastrophic Forgetting
• Theorem 1
• Theorem 2
• Lemma 1: Continual Noise Memorization
• Lemma 2: Enhanced Signal Learning
• Lemma 3: Amplified Noise Memorization
• Lemma 4: Continual Signal Learning
• Lemma 5
• proof : Proof of \ref{['lem:max_Gamma_task_k']}