Multi-Prompt with Depth Partitioned Cross-Modal Learning

TL;DR

The paper addresses the limited expressivity of single-prompt prompting for open-world visual concepts by introducing PMPO, a depth-partitioned, multi-prompt cross-modal learning framework built on frozen vision-language models. PMPO learns multiple prompts that are tied to different depths of the visual encoder, enabling each prompt to capture distinct hierarchical visual information, and it optionally leverages manually designed prompts as priors. Through extensive experiments on 11 datasets across base-new generalization, cross-dataset transfer, and domain generalization, PMPO achieves strong improvements in harmonic mean and unseen-class accuracy, outperforming several state-of-the-art prompting methods. The results demonstrate PMPO’s robustness and practical potential for adapting large-scale VLP models to diverse recognition tasks with limited data, while acknowledging higher computational costs and data requirements as avenues for further improvement.

Abstract

In recent years, soft prompt learning methods have been proposed to fine-tune large-scale vision-language pre-trained models for various downstream tasks. These methods typically combine learnable textual tokens with class tokens as input for models with frozen parameters. However, they often employ a single prompt to describe class contexts, failing to capture categories' diverse attributes adequately. This study introduces the Partitioned Multi-modal Prompt (PMPO), a multi-modal prompting technique that extends the soft prompt from a single learnable prompt to multiple prompts. Our method divides the visual encoder depths and connects learnable prompts to the separated visual depths, enabling different prompts to capture the hierarchical contextual depths of visual representations. Furthermore, to maximize the advantages of multi-prompt learning, we incorporate prior information from manually designed templates and learnable multi-prompts, thus improving the generalization capabilities of our approach. We evaluate the effectiveness of our approach on three challenging tasks: new class generalization, cross-dataset evaluation, and domain generalization. For instance, our method achieves a $79.28$ harmonic mean, averaged over 11 diverse image recognition datasets ($+7.62$ compared to CoOp), demonstrating significant competitiveness compared to state-of-the-art prompting methods.

Figures (6)

• Figure 1: PMPO outperforms previous state-of-the-art models on 9 out of 11 diverse recognition datasets for the new classes generalization task.
• Figure 2: Graphical illustration of different prompt learning methods. The image encoder and text encoder are pre-trained on large-scale image-text pairs and remain frozen during prompt tuning. This figure provides a brief comparison of single-prompt, multi-prompt, multi-modal prompt, and our proposed PMPO method.
• Figure 3: Key motivation of PMPO. We visualize the Score-CAM results for each prompt on two ImageNet samples. It is clear that different prompts indeed focus on distinct attributes of the classes. This observation highlights the importance of capturing diverse characteristics for effective recognition, as it allows the model to better understand various aspects of each class, leading to more accurate and robust predictions.
• Figure 4: Overview of the PMPO training pipeline: PMPO is based on the frozen CLIP. $N$ learnable prefix prompts establish bridge connections with Partitioned Transformer Blocks (PTB) enabling multi-prompts to learn hierarchical contextual visual representations. Within the PTB, Deep Visual Prompt Tuning is utilized for extracting visual embeddings. Furthermore, the ensemble of multi-prompts is computed within the feature embedding space, enhancing the overall effectiveness of the model.
• Figure 5: Ablation number of prompts