Continual Learning with Synthetic Boundary Experience Blending

TL;DR

This work addresses catastrophic forgetting in continual learning by introducing synthetic boundary data (SBD) generated via differential-privacy–inspired latent-space noise to regularize decision boundaries. It then proposes Experience Blending (EB), which jointly trains on stored exemplars and SBD through a dual-model aggregation mechanism that blends models trained on different data streams. Empirical results across CIFAR-10, CIFAR-100, and Tiny ImageNet show consistent, substantial improvements over strong baselines, confirming the effectiveness of enriching the feature space near decision boundaries. The approach is practical, incurring no inference-time overhead and offering robustness across varied memory budgets and data-distribution settings, with clear avenues for extending to task-free CL and other domains.

Abstract

Continual learning (CL) seeks to mitigate catastrophic forgetting when models are trained with sequential tasks. A common approach, experience replay (ER), stores past exemplars but only sparsely approximates the data distribution, yielding fragile and oversimplified decision boundaries. We address this limitation by introducing synthetic boundary data (SBD), generated via differential privacy: inspired noise into latent features to create boundary-adjacent representations that implicitly regularize decision boundaries. Building on this idea, we propose Experience Blending (EB), a framework that jointly trains on exemplars and SBD through a dual-model aggregation strategy. EB has two components: (1) latent-space noise injection to synthesize boundary data, and (2) end-to-end training that jointly leverages exemplars and SBD. Unlike standard experience replay, SBD enriches the feature space near decision boundaries, leading to more stable and robust continual learning. Extensive experiments on CIFAR-10, CIFAR-100, and Tiny ImageNet demonstrate consistent accuracy improvements of 10%, 6%, and 13%, respectively, over strong baselines.

Figures (10)

• Figure 1: Comparing our method with the ER method. We enhance the CL learning by memorizing the synthetic boundary data (SBD) to improve decision boundary robustness. The red dots and shaded region represent new task data and distribution.
• Figure 2: Experience blending: Our method aggregates the model $\mathcal{M}_{R\cup E}$ trained on the key sample memory $R$ and the SBD memory $E$, and the model $\mathcal{M}_{E}$ trained only on $E$. The model receives only an image during inference.
• Figure 3: The SBD is generated by passing inputs through an image encoder followed by an SA layer, where batch-wise noise is added to the SA layer's output. The networks with ice or fire symbols indicate that the model is trainable or fixed, respectively.
• Figure 4: The model $\mathcal{M}_{R\cup E}$ takes either$r$or$e$ as input, while $\mathcal{M}_{E}$ only takes $e$. $F_R$ and $F_E$ are shared shallow subnetworks that integrate the inputs, and $C$ is the shared classifier.
• Figure 5: Validation and test accuracy plots.