Cross-Class Feature Augmentation for Class Incremental Learning

TL;DR

This paper tackles catastrophic forgetting in class incremental learning by introducing Cross-Class Feature Augmentation (CCFA), a feature-space augmentation technique inspired by adversarial perturbations that exploits a previously learned classifier to augment old-task representations without changing architecture. CCFA perturbs current-task features toward target old classes, uses the old classifier to generate pseudo-labels, and trains on both original and augmented features within existing knowledge-distillation frameworks. The method yields consistent improvements on CIFAR-100 and ImageNet across multiple baselines, especially under extreme memory constraints, and is demonstrated to be compatible with other augmentations while adding only modest computational overhead. Overall, CCFA provides a practical, architecture-agnostic approach to stabilizing decision boundaries for old tasks in continual learning with limited exemplars, enhancing both stability and generalization.

Abstract

We propose a novel class incremental learning approach by incorporating a feature augmentation technique motivated by adversarial attacks. We employ a classifier learned in the past to complement training examples rather than simply play a role as a teacher for knowledge distillation towards subsequent models. The proposed approach has a unique perspective to utilize the previous knowledge in class incremental learning since it augments features of arbitrary target classes using examples in other classes via adversarial attacks on a previously learned classifier. By allowing the cross-class feature augmentations, each class in the old tasks conveniently populates samples in the feature space, which alleviates the collapse of the decision boundaries caused by sample deficiency for the previous tasks, especially when the number of stored exemplars is small. This idea can be easily incorporated into existing class incremental learning algorithms without any architecture modification. Extensive experiments on the standard benchmarks show that our method consistently outperforms existing class incremental learning methods by significant margins in various scenarios, especially under an environment with an extremely limited memory budget.

Figures (4)

• Figure 1: Overall class incremental learning framework with the proposed Cross-Class Feature Augmentation (CCFA). Our model minimizes classification loss $\mathcal{L}_{\text{cls}}$ on training examples in a mini-batch sampled from the union of current task dataset and a small set of exemplars from the previous tasks while minimizing the distillation loss $\mathcal{L}_{\text{dist}}$. To deal with the catastrophic forgetting issue induced by data imbalance between the previous and current tasks, we employ the CCFA to generate diverse features supporting the decision boundaries of the old classifier ${g}_{k-1}(\cdot)$.
• Figure 2: Illustration of Cross-Class Feature Augmentation (CCFA). CCFA perturbs a feature representation in a direction such that the perturbed feature crosses the decision boundary in the previous classifier to the target class, which is different from the original class, and complements the features for the target classes learned up to the previous stages.
• Figure 3: t-SNE of the features from the memory buffer of the random 10 classes after training initial stage (Memory Features), features generated by CCFA (Augmented Features) and features from randomly selected 1000 images from training dataset (Train-Data Features). By allowing CCFA, each class in the old tasks populates samples in the feature space, which alleviates the collapse of the decision boundaries caused by sample deficiency for the old tasks.
• Figure 4: CKA between the features from the feature extractors of the first and each incremental stage for PODNet douillard2020podnet and PODNet + CCFA after each residual layer.