Pre-trained Vision and Language Transformers Are Few-Shot Incremental Learners

TL;DR

Few-shot class incremental learning (FSCIL) is hampered by forgetting and overfitting when transferring base-session knowledge to incremental sessions. The authors propose PriViLege, a framework that leverages pre-trained Vision and Language Transformers with Pre-trained Knowledge Tuning (PKT), an entropy-based divergence loss (L_ED), and a semantic knowledge distillation loss (L_SKD), coupled with a prototype classifier, to preserve pre-trained knowledge while acquiring domain-specific knowledge. Across CIFAR-100, miniImageNet, and CUB200, PriViLege achieves large margins over state-of-the-art baselines, validating the effectiveness of incorporating vision-language pre-trained models for FSCIL. The approach is further shown to transfer to CLIP-based setups and benefits from targeted layer tuning and modulation prompts, indicating a promising direction for robust, scalable FSCIL with large pre-trained models.

Abstract

Few-Shot Class Incremental Learning (FSCIL) is a task that requires a model to learn new classes incrementally without forgetting when only a few samples for each class are given. FSCIL encounters two significant challenges: catastrophic forgetting and overfitting, and these challenges have driven prior studies to primarily rely on shallow models, such as ResNet-18. Even though their limited capacity can mitigate both forgetting and overfitting issues, it leads to inadequate knowledge transfer during few-shot incremental sessions. In this paper, we argue that large models such as vision and language transformers pre-trained on large datasets can be excellent few-shot incremental learners. To this end, we propose a novel FSCIL framework called PriViLege, Pre-trained Vision and Language transformers with prompting functions and knowledge distillation. Our framework effectively addresses the challenges of catastrophic forgetting and overfitting in large models through new pre-trained knowledge tuning (PKT) and two losses: entropy-based divergence loss and semantic knowledge distillation loss. Experimental results show that the proposed PriViLege significantly outperforms the existing state-of-the-art methods with a large margin, e.g., +9.38% in CUB200, +20.58% in CIFAR-100, and +13.36% in miniImageNet. Our implementation code is available at https://github.com/KHU-AGI/PriViLege.

Figures (9)

• Figure 1: 5-way 5-shot FSCIL experiments on CIFAR-100.
• Figure 2: Overall framework of our method. In the base session, the newly proposed PKT trains the B-Prompt, VL-Prompt, and selected layers in the pre-trained ViT. $\mathcal{L}_{ED}$ drives the vision token in VL-Prompt to enhance discriminative ability for better classification. $\mathcal{L}_{SKD}$ leverages language embeddings to provide semantic knowledge to the language token in VL-Prompt.
• Figure 3: CLIP performance on CIFAR-100. We compare our proposed methods with zero-shot performance of CLIP and CEC.
• Figure 4: Feature space visualization on CUB200 to validate the efficacy of $\mathcal{L}_{ED}$.
• Figure 5: Class-wise accuracy on CUB200 to compare the performance of with and without $\mathcal{L}_{SKD}$.