AmPLe: Supporting Vision-Language Models via Adaptive-Debiased Ensemble Multi-Prompt Learning

TL;DR

AmPLe addresses two biases in multi-prompt learning for vision-language models: model-prompt matching bias across different VLMs and sample-prompt matching bias from prompt-irrelevant content. It introduces a Hybrid Model-Prompt Ensemble (HMPE) combined with an Adaptive-Debiased Weight Generation (ADWG) module guided by prompt-relevant sample semantics and information-theoretic regularization, supported by causal analysis. Empirically, AmPLe yields consistent gains across base-to-novel, cross-dataset, and domain-generalization tasks, with statistical significance and favorable complexity trade-offs. The approach is plug-and-play and extends to CLIP-like and BLIP-like architectures, offering improved generalization under limited data scenarios.

Abstract

Multi-prompt learning methods have emerged as an effective approach for facilitating the rapid adaptation of vision-language models to downstream tasks with limited resources. Existing multi-prompt learning methods primarily focus on utilizing various meticulously designed prompts within a single foundation vision-language model to achieve superior performance. However, the overlooked model-prompt matching bias hinders the development of multi-prompt learning, i.e., the same prompt can convey different semantics across distinct vision-language models, such as CLIP-ViT-B/16 and CLIP-ViT-B/32, resulting in inconsistent predictions of identical prompt. To mitigate the impact of this bias on downstream tasks, we explore an ensemble learning approach to sufficiently aggregate the benefits of diverse predictions. Additionally, we further disclose the presence of sample-prompt matching bias, which originates from the prompt-irrelevant semantics encapsulated in the input samples. Thus, directly utilizing all information from the input samples for generating weights of ensemble learning can lead to suboptimal performance. In response, we extract prompt-relevant semantics from input samples by leveraging the guidance of the information theory-based analysis, adaptively calculating debiased ensemble weights. Overall, we propose Adaptive-Debiased Ensemble MultiPrompt Learning, abbreviated as AmPLe, to mitigate the two types of bias simultaneously. Extensive experiments on three representative tasks, i.e., generalization to novel classes, new target datasets, and unseen domain shifts, show that AmPLe can widely outperform existing methods. Theoretical validation from a causal perspective further supports the effectiveness of AmPLe.

Figures (13)

• Figure 1: An example of using multiple prompts to describe the Shiba Inu class.
• Figure 2: Zero-shot performance using multiple prompts and different CLIP models on the UCF101 dataset. In the histograms, P0 to P5 correspond to predictions obtained by applying a single prompt in a single VLM (CLIP-ViT-B/16 or CLIP-ViT-B/32), MP is the simple aggregation (sum and average) of P0 to P5's diverse predictions within the same VLM, and MMP aggregates the diverse predictions from P0 to P5 across both CLIP-ViT-B/16 and CLIP-ViT-B/32. The best result of MMP highlights the potential of combining multiple prompts and different VLMs to improve model performance.
• Figure 3: Comparison of attention maps for the Guacamole class in the Food101 dataset. Visualization (a) shows the attention map using Tuning Ensemble for image classification, demonstrating that the model attends to both the Guacamole object and irrelevant areas when using all information of the input image for ensemble weights generation. Visualization (b) shows the attention map when using prompt-relevant information from the input image for ensemble weights generation, illustrating that the model focuses primarily on the Guacamole object with minimal attention to irrelevant areas.
• Figure 4: The overall framework of AmPLe. The dashed blue box represents the hybrid model-prompt ensemble learning module, which aggregates diverse predictions from multiple prompts and models to mitigate model-prompt matching bias. The dashed red box represents the adaptive-debiased weight generation module, which extracts prompt-relevant semantics from images to compute debiased ensemble weights, thereby mitigating sample-prompt matching bias. During training, the weight generator $\mathcal{W}$ and the redundancy deprivation network $\mathcal{R}$ are optimized using $\mathcal{L}_{ce}^{weight}$, $\mathcal{L}_{kl}$, and $\mathcal{L}_{mutual}$.
• Figure 5: An example of prompting GPT-4 with the query template to generate visual descriptions for the sphynx class in the Oxford Pets dataset.