Multi-modal Attribute Prompting for Vision-Language Models

TL;DR

This work tackles the challenge of adapting pre-trained Vision-Language Models like CLIP to few-shot tasks by introducing Multi-modal Attribute Prompting (MAP), which jointly leverages textual attribute prompts, visual attribute prompts, and attribute-level alignment. MAP uses an Adaptive Visual Attribute Enhancement (AVAE) module to refine visual prompts guided by textual semantics and formulates cross-modal alignment as an Optimal Transport problem solved by the Sinkhorn algorithm, combining global and attribute-level scores for final predictions. The approach is validated across 11 datasets and multiple settings (base-to-novel, few-shot, domain generalization, and cross-dataset transfer), consistently outperforming strong baselines. By explicitly modeling visual attributes and aligning them at the attribute level, MAP enhances fine-grained perception and robustness to background noise, offering practical improvements for open-vocabulary recognition in limited-data scenarios.

Abstract

Pre-trained Vision-Language Models (VLMs), like CLIP, exhibit strong generalization ability to downstream tasks but struggle in few-shot scenarios. Existing prompting techniques primarily focus on global text and image representations, yet overlooking multi-modal attribute characteristics. This limitation hinders the model's ability to perceive fine-grained visual details and restricts its generalization ability to a broader range of unseen classes. To address this issue, we propose a Multi-modal Attribute Prompting method (MAP) by jointly exploring textual attribute prompting, visual attribute prompting, and attribute-level alignment. The proposed MAP enjoys several merits. First, we introduce learnable visual attribute prompts enhanced by textual attribute semantics to adaptively capture visual attributes for images from unknown categories, boosting fine-grained visual perception capabilities for CLIP. Second, the proposed attribute-level alignment complements the global alignment to enhance the robustness of cross-modal alignment for open-vocabulary objects. To our knowledge, this is the first work to establish cross-modal attribute-level alignment for CLIP-based few-shot adaptation. Extensive experimental results on 11 datasets demonstrate that our method performs favorably against state-of-the-art approaches.

Figures (10)

• Figure 1: (a) Conventional prompting methods use hand-crafted or learnable context in combination with the class name to construct the text prompt. (b) Recent methods introduce attribute descriptions to create text attribute prompts containing more semantic content. (c) Our method jointly explores multi-modal attributes and attribute-level alignment, enhancing fine-grained visual perception and achieving attribute-level alignment between images and text categories.
• Figure 2: (a) Moon Orchid and (b) Japanese Anemone exhibit strikingly similar overall appearances. Visual attributes play a crucial role in distinguishing between them, such as the central yellow stamens of Japanese Anemone.
• Figure 3: The architecture of our method: MAP leverages textual attribute descriptions to construct textual attribute prompts and incorporates learnable visual attribute prompts for capturing visual attributes. In the Adaptive Visual Attribute Enhancement module, initial visual attribute prompts are enhanced by textual attribute prompts via the attribute-aware cross-attention layer. The Multi-modal Attribute Alignment module calculates the similarity score between visual attributes and textual attributes with the optimal transport.
• Figure 4: Main results of few-shot image classification on 11 datasets. MAP consistently outperforms other CLIP adaptation methods across all datasets, demonstrating the strong few-shot adaptability of MAP.
• Figure 5: The average few-shot image classification results of more methods across 11 datasets.