Dual Modality Prompt Tuning for Vision-Language Pre-Trained Model

TL;DR

This work tackles adapting large vision-language pre-trained models to downstream tasks with limited data. It introduces Dual-modality Prompt Tuning (DPT), jointly learning text prompts and visual prompts, and enhances it with Class-Aware Visual Prompt Tuning (CAVPT) that uses cross-attention between text prompts and image patches to concentrate features on target concepts. Across 11 datasets and various few-shot settings, DPT (especially with CAVPT) consistently outperforms existing prompt-based methods and zero-shot CLIP, demonstrating strong generalization and robustness. The approach offers a practical, parameter-efficient path to leverages foundation VLPs for diverse classification tasks, with public code provided.

Abstract

With the emergence of large pre-trained vison-language model like CLIP, transferable representations can be adapted to a wide range of downstream tasks via prompt tuning. Prompt tuning tries to probe the beneficial information for downstream tasks from the general knowledge stored in the pre-trained model. A recently proposed method named Context Optimization (CoOp) introduces a set of learnable vectors as text prompt from the language side. However, tuning the text prompt alone can only adjust the synthesized "classifier", while the computed visual features of the image encoder can not be affected , thus leading to sub-optimal solutions. In this paper, we propose a novel Dual-modality Prompt Tuning (DPT) paradigm through learning text and visual prompts simultaneously. To make the final image feature concentrate more on the target visual concept, a Class-Aware Visual Prompt Tuning (CAVPT) scheme is further proposed in our DPT, where the class-aware visual prompt is generated dynamically by performing the cross attention between text prompts features and image patch token embeddings to encode both the downstream task-related information and visual instance information. Extensive experimental results on 11 datasets demonstrate the effectiveness and generalization ability of the proposed method. Our code is available in https://github.com/fanrena/DPT.

Figures (7)

• Figure 1: Visualization of the attention map of the image encoder. (a) Original Image. (b) Zero-Shot CLIP/CoOp. (c) Our DPT. The images are selected from OxfordPets and Caltech101. The GT annotated object is marked by a red box. Best viewed in color.
• Figure 2: The overall architecture of our proposed DPT method. It consists of three learnable components: text prompt, visual prompt and class-aware visual prompt generated from a Class-Aware Visual Prompt Tuning (CAVPT) generator module whose detailed architecture is illustrated in Fig. \ref{['figure_CAVPT']}.
• Figure 3: The detailed architecture of the proposed class-aware visual prompt tuning (CAVPT) generator module.
• Figure 4: Main results on the 11 datasets with 1,2,4,8,16 shots with ViT-B/32. Note that we also compare our methods with CPT yao2021cpt on average accuracy.
• Figure 5: Main results on the 11 datasets with 1,2,4,8,16 shots with ViT-B/16. Note that we also compare our methods with CPT yao2021cpt on average accuracy.