A Retrospect to Multi-prompt Learning across Vision and Language
Ziliang Chen, Xin Huang, Quanlong Guan, Liang Lin, Weiqi Luo
TL;DR
This work analyzes why multi-prompt learning can outperform single-prompt approaches in vision–language models and proposes EMPL, an energy-based framework that samples multiple prompts conditioned on image features to balance in-domain accuracy with open-vocabulary generalization. By framing prompts as an energy-based distribution and optimizing with a meta-learning objective, EMPL reduces cross-modal modality gaps and mitigates cross-domain and cross-dataset generalization challenges. The authors provide both theoretical justification (modality gap and non-identifiability) and extensive empirical validation across base-to-new, cross-domain, and cross-dataset tasks, showing consistent gains without adding parameter count to CLIP-style backbones. The approach offers a practical, scalable path to robust open-vocabulary vision–language understanding, with notable improvements in few-shot retrieval and transfer scenarios, albeit with higher inference cost that invites future efficiency-focused work.
Abstract
The vision community is undergoing the unprecedented progress with the emergence of Vision-Language Pretraining Models (VLMs). Prompt learning plays as the holy grail of accessing VLMs since it enables their fast adaptation to downstream tasks with limited resources. Whereas existing researches milling around single-prompt paradigms, rarely investigate the technical potential behind their multi-prompt learning counterparts. This paper aims to provide a principled retrospect for vision-language multi-prompt learning. We extend the recent constant modality gap phenomenon to learnable prompts and then, justify the superiority of vision-language transfer with multi-prompt augmentation, empirically and theoretically. In terms of this observation, we propose an Energy-based Multi-prompt Learning (EMPL) to generate multiple prompt embeddings by drawing instances from an energy-based distribution, which is implicitly defined by VLMs. So our EMPL is not only parameter-efficient but also rigorously lead to the balance between in-domain and out-of-domain open-vocabulary generalization. Comprehensive experiments have been conducted to justify our claims and the excellence of EMPL.