Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multimodal Parameter-Efficient Few-Shot Class Incremental Learning

Marco D'Alessandro, Alberto Alonso, Enrique Calabrés, Mikel Galar

TL;DR

This paper tackles Few-Shot Class Incremental Learning (FSCIL) by introducing Continual Parameter-Efficient CLIP (CPE-CLIP), a multimodal, prompt-based method that freezes the CLIP backbone and learns lightweight prompts for both language and vision encoders. By employing layer-specific language prompts (G-Prompt), vision prompts projected from language context, prompt accumulation in the vision branch, and a session-aware prompt regularization factor $\alpha_t$, the approach enables transfer across sessions while dramatically reducing the number of trainable parameters. Empirical results on CIFAR100, miniImageNet, and CUB200 show state-of-the-art performance with lower training cost and reduced forgetting (PD), demonstrating strong practical impact for continual learning in resource-constrained, real-world settings. Limitations include reliance on CLIP pretraining for vocabulary coverage and potential difficulties for tasks without clear image-label pairs, motivating future work to generalize the framework to broader modalities and longer incremental sequences.

Abstract

Few-Shot Class Incremental Learning (FSCIL) is a challenging continual learning task, where limited training examples are available during several learning sessions. To succeed in this task, it is necessary to avoid over-fitting new classes caused by biased distributions in the few-shot training sets. The general approach to address this issue involves enhancing the representational capability of a pre-defined backbone architecture by adding special modules for backward compatibility with older classes. However, this approach has not yet solved the dilemma of ensuring high classification accuracy over time while reducing the gap between the performance obtained on larger training sets and the smaller ones. In this work, we propose an alternative approach called Continual Parameter-Efficient CLIP (CPE-CLIP) to reduce the loss of information between different learning sessions. Instead of adapting additional modules to address information loss, we leverage the vast knowledge acquired by CLIP in large-scale pre-training and its effectiveness in generalizing to new concepts. Our approach is multimodal and parameter-efficient, relying on learnable prompts for both the language and vision encoders to enable transfer learning across sessions. We also introduce prompt regularization to improve performance and prevent forgetting. Our experimental results demonstrate that CPE-CLIP significantly improves FSCIL performance compared to state-of-the-art proposals while also drastically reducing the number of learnable parameters and training costs.

Multimodal Parameter-Efficient Few-Shot Class Incremental Learning

TL;DR

This paper tackles Few-Shot Class Incremental Learning (FSCIL) by introducing Continual Parameter-Efficient CLIP (CPE-CLIP), a multimodal, prompt-based method that freezes the CLIP backbone and learns lightweight prompts for both language and vision encoders. By employing layer-specific language prompts (G-Prompt), vision prompts projected from language context, prompt accumulation in the vision branch, and a session-aware prompt regularization factor , the approach enables transfer across sessions while dramatically reducing the number of trainable parameters. Empirical results on CIFAR100, miniImageNet, and CUB200 show state-of-the-art performance with lower training cost and reduced forgetting (PD), demonstrating strong practical impact for continual learning in resource-constrained, real-world settings. Limitations include reliance on CLIP pretraining for vocabulary coverage and potential difficulties for tasks without clear image-label pairs, motivating future work to generalize the framework to broader modalities and longer incremental sequences.

Abstract

Few-Shot Class Incremental Learning (FSCIL) is a challenging continual learning task, where limited training examples are available during several learning sessions. To succeed in this task, it is necessary to avoid over-fitting new classes caused by biased distributions in the few-shot training sets. The general approach to address this issue involves enhancing the representational capability of a pre-defined backbone architecture by adding special modules for backward compatibility with older classes. However, this approach has not yet solved the dilemma of ensuring high classification accuracy over time while reducing the gap between the performance obtained on larger training sets and the smaller ones. In this work, we propose an alternative approach called Continual Parameter-Efficient CLIP (CPE-CLIP) to reduce the loss of information between different learning sessions. Instead of adapting additional modules to address information loss, we leverage the vast knowledge acquired by CLIP in large-scale pre-training and its effectiveness in generalizing to new concepts. Our approach is multimodal and parameter-efficient, relying on learnable prompts for both the language and vision encoders to enable transfer learning across sessions. We also introduce prompt regularization to improve performance and prevent forgetting. Our experimental results demonstrate that CPE-CLIP significantly improves FSCIL performance compared to state-of-the-art proposals while also drastically reducing the number of learnable parameters and training costs.
Paper Structure (15 sections, 13 equations, 2 figures, 5 tables)

This paper contains 15 sections, 13 equations, 2 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Few-Shot Image Classification
  4. Incremental Learning
  5. Few-Shot Class-Incremental Learning
  6. Parameter-Efficient Learning
  7. Problem Formulation
  8. Method
  9. Experiment
  10. Evaluation Benchmarks
  11. Implementation Details
  12. Hyperparameter Tuning
  13. Comparison with state-of-the-art models
  14. Ablation Study
  15. Conclusions

Figures (2)

  • Figure 1: Summary of CPE-CLIP architecture and training process. The picture on the left describes the general structure of CPE-CLIP, where the G-Prompt contributes to generalizing task-invariant knowledge on the language encoder and is then projected to the vision encoder. Vision prompts are accumulated across subsequent layers, while a replacement strategy is used for the language encoder. The image on the right depicts the regularization process where an increase in the number of seen classes reduces parameter gradients by means of the scaling factor $\alpha_t$, for a given session $t$.
  • Figure 2: Ablation study depicting top-1 accuracy of 5-run simulations for the main model (Full Model), and three ablated versions where accumulation is removed from the vision branch (No Accumulation), no prompts are processed by the vision branch (No Vision Prompts), and no regularization is applied (No Regularization). Session 0 refers to base class.