Efficient Non-Exemplar Class-Incremental Learning with Retrospective Feature Synthesis

TL;DR

This work proposes a more efficient NECIL method that replaces prototypes with synthesized retrospective features for old classes and introduces a similarity-based feature compensation mechanism that integrates generated old class features with similar new class features to synthesize robust retrospective representations.

Abstract

Despite the outstanding performance in many individual tasks, deep neural networks suffer from catastrophic forgetting when learning from continuous data streams in real-world scenarios. Current Non-Exemplar Class-Incremental Learning (NECIL) methods mitigate forgetting by storing a single prototype per class, which serves to inject previous information when sequentially learning new classes. However, these stored prototypes or their augmented variants often fail to simultaneously capture spatial distribution diversity and precision needed for representing old classes. Moreover, as the model acquires new knowledge, these prototypes gradually become outdated, making them less effective. To overcome these limitations, we propose a more efficient NECIL method that replaces prototypes with synthesized retrospective features for old classes. Specifically, we model each old class's feature space using a multivariate Gaussian distribution and generate deep representations by sampling from high-likelihood regions. Additionally, we introduce a similarity-based feature compensation mechanism that integrates generated old class features with similar new class features to synthesize robust retrospective representations. These retrospective features are then incorporated into our incremental learning framework to preserve the decision boundaries of previous classes while learning new ones. Extensive experiments on CIFAR-100, TinyImageNet, and ImageNet-Subset demonstrate that our method significantly improves the efficiency of non-exemplar class-incremental learning and achieves state-of-the-art performance.

Figures (9)

• Figure 1: Idea illustration. (a) Baseline: Use the stored prototypes as the classification features of the old classes to rebuild the classification boundaries. (b) Gaussian Noise Aug ref_pass: Use random Gaussian noise to enhance the prototypes, but it introduces inaccurate features and overlaps between class distributions due to noise interference to the representation space. (c) Prototype Mixing ref_rrfe: Combine the prototypes of the old classes randomly linearly to form the hybrid prototypes, but these obtained prototypes lack authenticity. (d) Our Method: Model the representation space of each old class using a multivariate Gaussian distribution, then generate classification features for old classes by sampling from the high-likelihood regions, thereby better capturing the diversity and accuracy of the generated representations.
• Figure 2: Overall framework of our proposed method for NECIL. The new samples from the current task are augmented using rotation transformations. These augmented new class samples, along with synthesized retrospective features, are then utilized to optimize the incremental classification model to consolidate previously acquired knowledge while learning new classes. In the process of retrospective feature synthesis, we model the feature space of each old class using a multivariate Gaussian distribution and sample old class representations from high-likelihood regions. Additionally, we select the most similar new class features to complement the sampled old class features, thereby creating efficient retrospective representations of the old classes. "CE loss" represents the cross-entropy classification loss. "Stored Statistics" refers to the feature means and covariance matrices, which are calculated and stored after each task is learned. $\tilde{C}_{t}$ represents the classes within task $t$ after rotation-based label augmentation.
• Figure 3: Average classification accuracy of our method and state-of-the-art methods at each phase during incremental learning on CIFAR-100 and TinyImageNet.
• Figure 4: Average classification accuracy for the component effectiveness analysis of our method on CIFAR-100 and TinyImageNet.
• Figure 5: Visualization results. (a) MGS enhances the clustering of old classes in deep feature space. (b) SFC increases the discrimination between old and new classification features.