PLOT: Prompt Learning with Optimal Transport for Vision-Language Models

TL;DR

PLOT introduces a novel OT-based framework for learning multiple prompts to describe diverse category characteristics in vision-language models. By representing images with local visual feature maps and prompts as a set, and aligning them via Sinkhorn OT in a two-stage (inner OT optimization, outer prompt learning) process, the method achieves substantial improvements in few-shot recognition across 11 datasets. The approach provides interpretable transport plans and visualizations showing prompts attend to complementary image regions, and extends gracefully to CoCoOp. While zero-shot performance remains unchanged, PLOT offers a principled path toward fine-grained cross-modal alignment with practical gains and insights for future dynamic prompt design.

Abstract

With the increasing attention to large vision-language models such as CLIP, there has been a significant amount of effort dedicated to building efficient prompts. Unlike conventional methods of only learning one single prompt, we propose to learn multiple comprehensive prompts to describe diverse characteristics of categories such as intrinsic attributes or extrinsic contexts. However, directly matching each prompt to the same visual feature is problematic, as it pushes the prompts to converge to one point. To solve this problem, we propose to apply optimal transport to match the vision and text modalities. Specifically, we first model images and the categories with visual and textual feature sets. Then, we apply a two-stage optimization strategy to learn the prompts. In the inner loop, we optimize the optimal transport distance to align visual features and prompts by the Sinkhorn algorithm, while in the outer loop, we learn the prompts by this distance from the supervised data. Extensive experiments are conducted on the few-shot recognition task and the improvement demonstrates the superiority of our method. The code is available at https://github.com/CHENGY12/PLOT.

Figures (6)

• Figure 1: The motivation that one category can be complementarily described in different views (An example of "Brambling").
• Figure 2: The framework:PLOT first describes each category with multiple prompts and obtains a set of prompt features by text encoder. The image is also encoded as a set of local features. Then the optimal transport is used as the metric between prompts and visual features.
• Figure 3: The few-shot learning results on 11 datasets. We compare our PLOT with CoOp, CoCoOp, and the Linear Probe method and observe the consistent and significant performance improvement on most datasets. (The average accuracy on all datasets is shown on the left top.)
• Figure 4: Visualization. We provide the heatmaps of transport plan $\bm{T}$ related to each prompt on 4 categories in ImageNet. Different transport plans focus on different attributes of the object.
• Figure A1: Failure Visualization. We provide the heatmaps of transport plan T related to each prompt on 2 failure examples in the StanfordCars dataset.