Prompt Tuning Vision Language Models with Margin Regularizer for Few-Shot Learning under Distribution Shifts

TL;DR

This work tackles adapting large vision-language models (e.g., CLIP) to target datasets with very limited labels and distribution shifts, without access to source data. It introduces PromptMargin, a completely source-free prompt-tuning framework built on two modules: Selective Augmentation, which curates augmentations to augment scarce data efficiently, and Multimodal Margin Regularization (MMReg), which enforces uniform inter-class separation in both text and image spaces and aligns them via a joint loss $\mathcal{L}_{total} = \mathcal{L}_{CE} + \mathcal{R}(\tilde{X_T}) + \mathcal{R}(\tilde{X_V})$. The method demonstrates strong performance across fifteen diverse target benchmarks in 1-shot and 5-shot settings, frequently outperforming MaPLe and other baselines, especially when class names are unavailable or semantically weak. This shows that large-scale vision-language models can be effectively and efficiently adapted to challenging real-world tasks without source-domain data, broadening their applicability to specialized domains.

Abstract

Recently, Vision-Language foundation models like CLIP and ALIGN, which are pre-trained on large-scale data have shown remarkable zero-shot generalization to diverse datasets with different classes and even domains. In this work, we take a step further and analyze whether these models can be adapted to target datasets having very different distributions and classes compared to what these models have been trained on, using only a few labeled examples from the target dataset. In such scenarios, finetuning large pretrained models is challenging due to problems of overfitting as well as loss of generalization, and has not been well explored in prior literature. Since, the pre-training data of such models are unavailable, it is difficult to comprehend the performance on various downstream datasets. First, we try to answer the question: Given a target dataset with a few labelled examples, can we estimate whether further fine-tuning can enhance the performance compared to zero-shot evaluation? by analyzing the common vision-language embedding space. Based on the analysis, we propose a novel prompt-tuning method, PromptMargin for adapting such large-scale VLMs directly on the few target samples. PromptMargin effectively tunes the text as well as visual prompts for this task, and has two main modules: 1) Firstly, we use a selective augmentation strategy to complement the few training samples in each task; 2) Additionally, to ensure robust training in the presence of unfamiliar class names, we increase the inter-class margin for improved class discrimination using a novel Multimodal Margin Regularizer. Extensive experiments and analysis across fifteen target benchmark datasets, with varying degrees of distribution shifts from natural images, shows the effectiveness of the proposed framework over the existing state-of-the-art approaches applied to this setting. github.com/debarshigit/PromptMargin.

Figures (8)

• Figure 1: An overview of our proposed PromptMargin framework. A randomly sampled episode from the target dataset is considered. The support set images along with their augmentations are passed through the CLIP image encoder, and their labels are passed though the CLIP text encoder. The selective augmentation strategy selects augmentations based on the embedding vectors. The Max-Margin Regularizer (MMReg) enforces the class-wise image prototypes and the text embeddings to uniformly separate out.
• Figure 2: Effectiveness of the MMReg. All the heatmaps represent inter-class $L_2$ distances between embeddings for a representative episode in the 5-way setting. Darker hues represent higher values and vice versa. In all the images, the text features are followed by image features from left to right. (a) represents initial text and image feature distances, (b) represents the text and image embedding distances without MMReg, (c) represents the text and image distances with only text regularization, while (d) represents the text and image embeddings with MMReg. We observe in (b) that there is significant difference between the interclass distance of the image and text embeddings, implying that their embeddings are not that similar. Using only text regularization part separates the text but not the image features. In contrast, the maps are very similar for (d), which justifies the usefulness of the MMReg.
• Figure 3: Some qualitative results across different datasets. From top left, samples are shown from Quickdraw quickdraw, EuroSAT eurosat, ISIC isic and Fungi fungi datasets. Green and red denote correct and incorrect predictions respectively.
• Figure 4: Some instances of poor augmentations generated for two support set images, leading to reduced generalization. (a) and (b) represents images from two classes of the Aircraft dataset, namely "Hawk T1" and "Boeing 747-400".
• Figure 5: Visualizations of the augmentations being selected by the proposed Selective Augmentation module for EuroSAT and ISIC with 20 initial augmentations. Each row corresponds to a particular class. The red borders denote the ones which are discarded. We observe that augmentations where the region of interest are removed or darkened, have fairly more chance of getting discarded. However, in some cases, more augmentations may be needed to be removed than only five.