Prototype-Based Continual Learning with Label-free Replay Buffer and Cluster Preservation Loss

TL;DR

This paper tackles catastrophic forgetting in continual learning by introducing a label-free replay buffer coupled with a cluster preservation loss and specialized contrastive mechanisms. The proposed iSL-LRCP (supervised) and iUL-LRCP (unsupervised) leverage prototype-based classification, computed via K-means, and store class prototypes with support samples in a compact, privacy-preserving buffer. A trifecta of losses—supervised contrastive, cluster preservation, and push-away/pull-toward—enables robust retention of prior structure while accommodating class- or domain-shifts, with an unsupervised variant achieving competitive results. Across CI and DI benchmarks, the method consistently surpasses strong replay-based and replay-free baselines, validating the effectiveness of preserving latent cluster structure for continual adaptation.

Abstract

Continual learning techniques employ simple replay sample selection processes and use them during subsequent tasks. Typically, they rely on labeled data. In this paper, we depart from this by automatically selecting prototypes stored without labels, preserving cluster structures in the latent space across tasks. By eliminating label dependence in the replay buffer and introducing cluster preservation loss, it is demonstrated that the proposed method can maintain essential information from previously encountered tasks while ensuring adaptation to new tasks. "Push-away" and "pull-toward" mechanisms over previously learned prototypes are also introduced for class-incremental and domain-incremental scenarios. These mechanisms ensure the retention of previously learned information as well as adaptation to new classes or domain shifts. The proposed method is evaluated on several benchmarks, including SplitCIFAR100, SplitImageNet32, SplitTinyImageNet, and SplitCaltech256 for class-incremental, as well as R-MNIST and CORe50 for domain-incremental setting using pre-extracted DINOv2 features. Experimental results indicate that the label-free replay-based technique outperforms state-of-the-art continual learning methods and, in some cases, even surpasses offline learning. An unsupervised variant of the proposed technique for the class-incremental setting, avoiding labels use even on incoming data, also demonstrated competitive performance, outperforming particular supervised baselines in some cases. These findings underscore the effectiveness of the proposed framework in retaining prior information and facilitating continual adaptation.

Figures (6)

• Figure 1: Overview of the continual learning framework. Features are extracted using a pre-trained DINOv2 ViT-L/14 backbone and projected into a 512-dimensional space. Class prototypes are computed with K-means and support samples are selected via $\sigma$-band sampling, and then stored in a label-free replay buffer.
• Figure 2: Illustration of the Cluster Preservation Loss. This mechanism retains the structural integrity of clusters by minimizing distribution shifts of class prototypes and support samples in the latent space across tasks. Dashed lines indicate the $\sigma$-bands within which support samples are selected, ensuring consistency and preventing catastrophic forgetting.
• Figure 3: The combined Supervised Contrastive and Push-Away Loss. The supervised contrastive loss ensures intra-class compactness and inter-class separation, while the push-away loss enforces additional separation between latent representations of new data points and prototypes from previously learned tasks, mitigating interference.
• Figure 4: Illustration of the combined Supervised Contrastive and Pull-Toward Loss. The supervised contrastive loss ensures intra-class compactness and inter-class separation within the current domain, while the pull-toward loss aligns representations of new domain samples with prototypes from the first domain, promoting consistency across domains and reducing domain shifts.
• Figure 5: Comparison of accuracy and forgetting scores for different methods across CI datasets. (a) SplitCIFAR100; (b) SplitTinyImageNet.