Few-shot Class-Incremental Learning via Generative Co-Memory Regularization

TL;DR

This work proposes a generative co-memory regularization approach to facilitate FSCIL, which improves recognition accuracy, while mitigating catastrophic forgetting over old classes and overfitting to novel classes.

Abstract

Few-shot class-incremental learning (FSCIL) aims to incrementally learn models from a small amount of novel data, which requires strong representation and adaptation ability of models learned under few-example supervision to avoid catastrophic forgetting on old classes and overfitting to novel classes. This work proposes a generative co-memory regularization approach to facilitate FSCIL. In the approach, the base learning leverages generative domain adaptation finetuning to finetune a pretrained generative encoder on a few examples of base classes by jointly incorporating a masked autoencoder (MAE) decoder for feature reconstruction and a fully-connected classifier for feature classification, which enables the model to efficiently capture general and adaptable representations. Using the finetuned encoder and learned classifier, we construct two class-wise memories: representation memory for storing the mean features for each class, and weight memory for storing the classifier weights. After that, the memory-regularized incremental learning is performed to train the classifier dynamically on the examples of few-shot classes in each incremental session by simultaneously optimizing feature classification and co-memory regularization. The memories are updated in a class-incremental manner and they collaboratively regularize the incremental learning. In this way, the learned models improve recognition accuracy, while mitigating catastrophic forgetting over old classes and overfitting to novel classes. Extensive experiments on popular benchmarks clearly demonstrate that our approach outperforms the state-of-the-arts.

Figures (7)

• Figure 1: We apply co-memory regularization in model learning, which facilitates discriminative classification while mitigating catastrophic forgetting and overfitting.
• Figure 2: The framework of our approach consists of two steps. The generative domain adaptation finetuning step finetunes a pretrained ViT encoder on the base classes by jointly incorporating a MAE decoder for self-supervised masked feature reconstruction and a two-layer classifier for supervised feature classification. Then, the ViT encoder is frozen and used to construct two cooperative memories with image examples: representation memory which stores the class mean features over each class, and weight memory which stores the classifier weights of two fully-connected layers before the final softmax layer. Finally, the memory-regularized incremental learning step updates two memories and uses them to collaboratively regularize the incremental learning in each session explicitly and implicitly, respectively.
• Figure 3: The base accuracy on MiniImageNet with different pretraining datasets. Here, the model pretrained on ImageNet achieves the best accuracy but leaks information.
• Figure 4: The effect of finetuning in incremental sessions on CIFAR100.
• Figure 5: t-SNE representation visualization. We randomly select the examples over several base and incremental classes to show how these classes change at base (top), intermediate incremental (middle) and last incremental (bottom) sessions. '$\bullet$', '$\blacktriangle$' and '$\bigstar$' represent base classes, incremental classes and class mean features, respectively.