Modeling Variants of Prompts for Vision-Language Models

TL;DR

This paper tackles the sensitivity of vision-language models to prompt templates by introducing RobustPrompt Benchmark and MVP. RobustPrompt Benchmark offers a six-type taxonomy, a 733-template robust dataset, and a Prompt Robustness Score to quantify cross-template stability. MVP decouples templates from class names and uses a Variational Autoencoder to model the distribution of diverse prompt structures, training with a multi-template loss and a VAE loss, and achieving superior robustness across 11 datasets while maintaining strong few-shot performance. The results suggest that modeling prompt variation with a probabilistic, human-readable approach can substantially mitigate prompt sensitivity in VLMs, enabling more reliable deployment across diverse natural language prompts.

Abstract

Large pre-trained vision-language models (VLMs) offer a promising approach to leveraging human language for enhancing downstream tasks. However, VLMs such as CLIP face significant limitation: its performance is highly sensitive to prompt template design. Although prompt learning methods can address the sensitivity issue by replacing natural language prompts with learnable ones, they are incomprehensible to humans. Ensuring consistent performance across various prompt templates enables models to adapt seamlessly to diverse phrasings, enhancing their ability to handle downstream tasks without requiring extensive prompt engineering. In this work, we introduce the RobustPrompt Benchmark, a systematic benchmark to evaluate robustness to different prompt templates for VLMs. It includes a dataset with hundreds of carefully designed prompt templates, divided into six types, covering a wide variety of commonly used templates. Beside the benchmark, we propose Modeling Variants of Prompts (MVP), a simple yet effective method that mitigates sensitivity by modeling variants of prompt structures. The innovation of MVP lies in decoupling prompts into templates and class names, and using Variational Autoencoders (VAE) to model the distribution of diverse prompt structures. Experiments across 11 datasets demonstrate that MVP can greatly enhance model robustness to variations in input prompts without a drop in performance. The code is available at https://github.com/liaolea/MVP.

Figures (7)

• Figure 1: (a) Robustness comparison of zero-shot CLIP and our method on the EuroSAT dataset. The performance of zero-shot CLIP varies significantly with changes in the prompt template. In contrast, our method maintains strong robustness across different prompt templates. (b) T-SNE visualization of text features. We conduct a T-SNE analysis using 10 classes from the EuroSAT dataset and 300 different prompt templates to visualize the embeddings in the feature space. The text features generated by our method exhibit larger inter-class distances and reduced intra-class variation.
• Figure 2: Impact of different prompt templates on zero-shot CLIP. This figure illustrates the performance variations across different prompt template types, highlighting their influence on model robustness.
• Figure 3: The MVP Architecture. MVP primarily consists of CLIP, VAE, and a fusion block. (1) During each training epoch, we randomly select a subset of templates from the robust prompt dataset. We naturally decouple the original prompt into two components: prompt templates and class names, which are then processed separately by the CLIP text encoder. The template component is processed by the VAE to model the distribution of templates, while the class names are directly encoded. Subsequently, features from both components are concatenated and integrated through the fusion block. Finally, the resulting fused representation is used to compute cosine similarity with image features for classification. (2) During inference, only a single prompt template is provided as input.
• Figure 4: Main results of accuracy on 11 datasets. MVP outperforms zero-shot CLIP, CoOp, and CLIP-Adapter in overall accuracy, while demonstrating competitive performance with Tip-Adapter-F, showing strengths and weaknesses across different datasets.
• Figure 5: Visualization of the L2 distance and cosine similarity between the text features generated by CLIP text encoder and the text features reconstructed by the VAE on the ImageNet dataset.