Incremental Learning with Repetition via Pseudo-Feature Projection

TL;DR

This work addresses exemplar-free continual learning under realistic data streams with class repetition by introducing repetition-based CIFAR-50/10 scenarios and benchmarking a broad set of methods. It proposes Horde, an ensemble-based approach that grows with a budget of $B$ self-reliant feature extractors, and uses pseudo-feature projection to align representations via class prototypes. Across three repetition scenarios, Horde delivers state-of-the-art performance in settings with repetition while remaining competitive in no-repetition cases, highlighting the practical value of dynamic feature spaces and replay via prototypes. The findings underscore the importance of stability-plasticity management in realistic continual learning, with implications for privacy-preserving, edge-deployed systems that encounter recurring concepts.

Abstract

Incremental Learning scenarios do not always represent real-world inference use-cases, which tend to have less strict task boundaries, and exhibit repetition of common classes and concepts in their continual data stream. To better represent these use-cases, new scenarios with partial repetition and mixing of tasks are proposed, where the repetition patterns are innate to the scenario and unknown to the strategy. We investigate how exemplar-free incremental learning strategies are affected by data repetition, and we adapt a series of state-of-the-art approaches to analyse and fairly compare them under both settings. Further, we also propose a novel method (Horde), able to dynamically adjust an ensemble of self-reliant feature extractors, and align them by exploiting class repetition. Our proposed exemplar-free method achieves competitive results in the classic scenario without repetition, and state-of-the-art performance in the one with repetition.

Figures (12)

• Figure 1: Overview of our proposed method (Horde). Each data sample is processed by an ensemble of independent feature extractors. The features from all extractors are concatenated before being passed into a unified head that can accommodate the dynamic input size through pseudo-feature projection.
• Figure 2: Overview of the steps our proposed method (Horde) performs for each incremental task.
• Figure 3: Accuracy curves for scenario (b) with equiprobable repetition frequency. Weight-regularized methods (solid) benefit directly from short tasks with class repetition, while prototype-based approaches (dashed) degrade in accuracy as the sequence advances.
• Figure 4: Depiction of the task accuracy progression of MAS over the scenario (b) sequence (averaged over 5 seeds). Accuracy is evaluated on the test set for the classes represented in the corresponding incremental training data within a task. Note, that for repetition there is always a certain overlap within tasks.
• Figure 5: Accuracy results for finetuning and weight-regularization based methods. Solid lines indicate the backpropagation of the cross-entropy loss over all classes leading to catastrophic class recency bias. Dashed lines indicate the freezing of the weights related to the output of non-current classes.