Few Shot Class Incremental Learning using Vision-Language models

TL;DR

The paper tackles few-shot class incremental learning by bridging image and text semantics through a language regularizer and a semantic subspace regularizer, enabling Vision-Language models to inform base training and stabilize incremental updates. By incorporating a cross-domain graph Laplacian regularization and convex-combination-based regularization for new classes, the approach mitigates catastrophic forgetting while enabling rapid adaptation to novel classes with limited samples. Empirical results across miniImageNet, CIFAR-100, and tieredImageNet demonstrate state-of-the-art performance in multi-session and competitive performance in single-session settings, with CLIP-based semantic representations and Top-K cosine similarity yielding notable gains. The work highlights the value of leveraging language-derived semantics in FSCIL and points to promising directions in prompt design and broader VLM integration for continual learning tasks.

Abstract

Recent advancements in deep learning have demonstrated remarkable performance comparable to human capabilities across various supervised computer vision tasks. However, the prevalent assumption of having an extensive pool of training data encompassing all classes prior to model training often diverges from real-world scenarios, where limited data availability for novel classes is the norm. The challenge emerges in seamlessly integrating new classes with few samples into the training data, demanding the model to adeptly accommodate these additions without compromising its performance on base classes. To address this exigency, the research community has introduced several solutions under the realm of few-shot class incremental learning (FSCIL). In this study, we introduce an innovative FSCIL framework that utilizes language regularizer and subspace regularizer. During base training, the language regularizer helps incorporate semantic information extracted from a Vision-Language model. The subspace regularizer helps in facilitating the model's acquisition of nuanced connections between image and text semantics inherent to base classes during incremental training. Our proposed framework not only empowers the model to embrace novel classes with limited data, but also ensures the preservation of performance on base classes. To substantiate the efficacy of our approach, we conduct comprehensive experiments on three distinct FSCIL benchmarks, where our framework attains state-of-the-art performance.

Figures (1)

• Figure 1: Overview of our few-shot incremental learning framework. (a) For base model training, we use cross-entropy loss and language regularizer loss. (b) For incremental training, Semantic Subspace Regularizer loss and fine-tuning regularizer loss are used. Weight regularization for backbone and classifier is ignored in the diagram for simplicity.