Conditional Prototype Rectification Prompt Learning

TL;DR

The paper addresses overfitting and data-inefficiency in efficient transfer learning for vision-language models by introducing Conditional Prototype Rectification Prompt Learning (CPR). CPR fuses visual and textual prototypes through a Conditional Adapter (CoAdapter) and refines prototypes with Nearest Neighbor Rectification (NNR) using unlabeled data, guided by a consistency constraint that aligns fused representations with text-enhanced features from an LLM. The approach yields state-of-the-art performance on 11 datasets in both few-shot classification and base-to-new generalization tasks, while keeping tuning lightweight by updating only a small adapter and transformer block. This work advances data-efficient, multi-modality tuning for VLMs, enabling robust generalization with limited labeled data and without heavy data augmentation.

Abstract

Pre-trained large-scale vision-language models (VLMs) have acquired profound understanding of general visual concepts. Recent advancements in efficient transfer learning (ETL) have shown remarkable success in fine-tuning VLMs within the scenario of limited data, introducing only a few parameters to harness task-specific insights from VLMs. Despite significant progress, current leading ETL methods tend to overfit the narrow distributions of base classes seen during training and encounter two primary challenges: (i) only utilizing uni-modal information to modeling task-specific knowledge; and (ii) using costly and time-consuming methods to supplement knowledge. To address these issues, we propose a Conditional Prototype Rectification Prompt Learning (CPR) method to correct the bias of base examples and augment limited data in an effective way. Specifically, we alleviate overfitting on base classes from two aspects. First, each input image acquires knowledge from both textual and visual prototypes, and then generates sample-conditional text tokens. Second, we extract utilizable knowledge from unlabeled data to further refine the prototypes. These two strategies mitigate biases stemming from base classes, yielding a more effective classifier. Extensive experiments on 11 benchmark datasets show that our CPR achieves state-of-the-art performance on both few-shot classification and base-to-new generalization tasks. Our code is avaliable at \url{https://github.com/chenhaoxing/CPR}.

Figures (4)

• Figure 1: Comparison of different efficient transfer learning methods on base-to-new generalization task.
• Figure 2: Our approach, Conditional Prototype Rectification Prompt Learning (CPR), includes two strategies: (a) Conditional Adapter exploits connections between input images and both visual and textual prototypes, integrating textual and visual structures to model task-specific knowledge. (b) Nearest Neighbor Rectification leverages unlabeled data to avoid external or synthetic data, addressing biases and limitations in few-shot learning.
• Figure 3: The performance comparison for few-shot learning (1-/2-/4-/8-/16-shot) is conducted on 11 benchmark datasets, with the top-left indicating the averaged accuracy across these datasets.
• Figure 4: The ablation studies for two coefficients $\alpha$ and $k$ on FGVCAircraft under few-shot learning setting.