GalLoP: Learning Global and Local Prompts for Vision-Language Models

TL;DR

GalLoP addresses the robustness-accuracy trade-off in vision-language prompt learning by jointly training diverse global prompts and sparsely aligned local prompts. It introduces a sparse, scale-aware local alignment with a lightweight linear projector to improve text-vision correspondence, and enforces diversity via prompt dropout and multiscale local prompts. Empirical results across 11 datasets and robustness benchmarks show GalLoP achieves state-of-the-art or competitive top-1 accuracy while delivering superior domain generalization and OOD detection performance, outperforming several dedicated robustness methods. The work suggests that combining strong local discriminability with diverse global-local representations yields substantial practical gains for adapting VLMs in few-shot settings.

Abstract

Prompt learning has been widely adopted to efficiently adapt vision-language models (VLMs), e.g. CLIP, for few-shot image classification. Despite their success, most prompt learning methods trade-off between classification accuracy and robustness, e.g. in domain generalization or out-of-distribution (OOD) detection. In this work, we introduce Global-Local Prompts (GalLoP), a new prompt learning method that learns multiple diverse prompts leveraging both global and local visual features. The training of the local prompts relies on local features with an enhanced vision-text alignment. To focus only on pertinent features, this local alignment is coupled with a sparsity strategy in the selection of the local features. We enforce diversity on the set of prompts using a new ``prompt dropout'' technique and a multiscale strategy on the local prompts. GalLoP outperforms previous prompt learning methods on accuracy on eleven datasets in different few shots settings and with various backbones. Furthermore, GalLoP shows strong robustness performances in both domain generalization and OOD detection, even outperforming dedicated OOD detection methods. Code and instructions to reproduce our results: https://github.com/MarcLafon/gallop.

Figures (13)

• Figure 1: Our GalLoP method demonstrates excellent performances in accuracy plus robustness, i.e. out-of-distribution detection (a) and domain generalization (b), while state-of-the-art prompt learning methods compromise between these aspects. Additionally, unlike recent methods utilizing ineffective local zero-shot CLIP features, GalLoP learns discriminative local prompts precisely aligned with sparse image regions at various scales, facilitating the discriminability between classes. GalLoP integrates both global and local prompts, with their diversity explicitly enforced during few-shot learning, which significantly enhances the performance of their combination (c).
• Figure 2: Illustration of GalLoP. GalLoP learns a diverse set of global prompts and local prompts. Pertinent local prompts are learned using only the most relevant regions of the image for each class. We further improve the limited text-vision alignment of CLIP's local features using a simple linear layer. The diversity is encouraged using a new "prompt dropout" technique for global prompts, and a multiscale loss for local prompts.
• Figure 3: GalLoP sparse local similarity$\text{sim}(\mathcal{Z}_l,~ \bm{t}_c)$ between class prompt $\bm{t}_c$ and visual features $\mathcal{Z}_l$ is the average of the top-$k$ highest similarities (here, k=3).
• Figure 4: (a) Prompt dropout induces diversity by randomly selecting different subsets of prompts for each image of the batch. In (a), each image will be used by half the prompts. (b) To learn diverse local prompts, we specialize each one of them using a different number of regions, and therefore a different level of sparsity.
• Figure 5: Results on ImageNet with different few shot settings \ref{['fig:main_few_shot_imagenet']}, and ResNet-50 \ref{['fig:backbone_bar_plot']}.