Covariance-based Space Regularization for Few-shot Class Incremental Learning

TL;DR

This paper proposes a simple yet effective covariance constraint loss to force the model to learn each class distribution with the same covariance matrix, and proposes a perturbation approach to perturb the few-shot training samples in the feature space, which encourages the samples to be away from the weighted distribution of other classes.

Abstract

Few-shot Class Incremental Learning (FSCIL) presents a challenging yet realistic scenario, which requires the model to continually learn new classes with limited labeled data (i.e., incremental sessions) while retaining knowledge of previously learned base classes (i.e., base sessions). Due to the limited data in incremental sessions, models are prone to overfitting new classes and suffering catastrophic forgetting of base classes. To tackle these issues, recent advancements resort to prototype-based approaches to constrain the base class distribution and learn discriminative representations of new classes. Despite the progress, the limited data issue still induces ill-divided feature space, leading the model to confuse the new class with old classes or fail to facilitate good separation among new classes. In this paper, we aim to mitigate these issues by directly constraining the span of each class distribution from a covariance perspective. In detail, we propose a simple yet effective covariance constraint loss to force the model to learn each class distribution with the same covariance matrix. In addition, we propose a perturbation approach to perturb the few-shot training samples in the feature space, which encourages the samples to be away from the weighted distribution of other classes. Regarding perturbed samples as new class data, the classifier is forced to establish explicit boundaries between each new class and the existing ones. Our approach is easy to integrate into existing FSCIL approaches to boost performance. Experiments on three benchmarks validate the effectiveness of our approach, achieving a new state-of-the-art performance of FSCIL.

Figures (5)

• Figure 1: (a) Prototype-based models demonstrate compact representations of old classes, conserving space for new classes, though risking confusion due to mixed class distributions. (b) The proposed approach aims to regularize each class distribution within a fixed span by constraining covariance and to enhance class separation through the learning of perturbed new class data.
• Figure 2: (a) Base session training. We deploy the covariance constraint loss to learn class distributions with a fixed span. (b) Semantic perturbation learning for the incremental stages. New data samples are perturbed by multiplying with the predicted distribution. The perturbed samples are trained along with the original data to establish the separation between classes.
• Figure 3: (a) Comparison of harmonic performance after incremental sessions with baseline models on MiniImageNet. (b) Comparison of performance of new classes with baseline model in each incremental session on MiniImageNet
• Figure 4: Comparisons of T-SNE visualization of the learned feature embedding on CIFAR100. (a) Visualization of eight base classes' feature embedding in the base session. We compare the base model with and without covariance constraint loss. (b) Visualizations of the feature embedding of two few-shot new classes together with their perturbed samples, and three base classes during incremental learning. We compare the baseline model and the model using the proposed SPL.
• Figure 5: (a) Results of different incremental shots on CUB200. (b) Results of different hyperparameters on CUB200.