Consistency-guided Prompt Learning for Vision-Language Models
Shuvendu Roy, Ali Etemad
TL;DR
CoPrompt tackles the problem of fine-tuning vision-language foundation models in few-shot settings without sacrificing zero-shot generalization. It introduces a cosine-based consistency constraint between trainable prompts/adapters and frozen pre-trained encoders, augmented by perturbations in text (via an LLM) and image augmentations, and combines prompting and adapter modules. The method yields state-of-the-art results on base-to-novel generalization, cross-dataset evaluation, and domain generalization across 11 datasets, with extensive ablations validating each component. This approach offers practical improvements for adapting large vision-language models with limited data, balancing learning capacity and generalization.
Abstract
We propose Consistency-guided Prompt learning (CoPrompt), a new fine-tuning method for vision-language models. Our approach improves the generalization of large foundation models when fine-tuned on downstream tasks in a few-shot setting. The basic idea of CoPrompt is to enforce a consistency constraint in the prediction of the trainable and pre-trained models to prevent overfitting on the downstream task. Additionally, we introduce the following two components into our consistency constraint to further boost the performance: enforcing consistency on two perturbed inputs and combining two dominant paradigms of tuning, prompting and adapter. Enforcing consistency on perturbed input serves to further regularize the consistency constraint, thereby improving generalization. Moreover, the integration of adapters and prompts not only enhances performance on downstream tasks but also offers increased tuning flexibility in both input and output spaces. This facilitates more effective adaptation to downstream tasks in a few-shot learning setting. Experiments show that CoPrompt outperforms existing methods on a range of evaluation suites, including base-to-novel generalization, domain generalization, and cross-dataset evaluation. On generalization, CoPrompt improves the state-of-the-art on zero-shot tasks and the overall harmonic mean over 11 datasets. Detailed ablation studies show the effectiveness of each of the components in CoPrompt. We make our code available at https://github.com/ShuvenduRoy/CoPrompt.