Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Diff-Prompt: Diffusion-Driven Prompt Generator with Mask Supervision

Weicai Yan, Wang Lin, Zirun Guo, Ye Wang, Fangming Feng, Xiaoda Yang, Zehan Wang, Tao Jin

TL;DR

Diff-Prompt tackles the limited gains of prompt learning for fine-grained multimodal tasks by using diffusion in latent space to generate rich, input-specific prompts. It introduces a three-stage pipeline: (1) Mask-VAE compresses masks into a latent representation, (2) a diffusion-based prompt generator creates prompts conditioned on image and caption, and (3) adapters align the generated prompts with the foundation model for fine-tuning, complemented by global prompts. Empirically, Diff-Prompt yields consistent improvements on RefCOCO and Flickr30k over various adapter/prompt baselines, demonstrates deeper prompt interactions enhancing performance, and shows promising cross-dataset/domain generalization and zero-shot capabilities. The approach underscores the potential of generative priors for prompt synthesis, enabling richer cross-modal guidance while maintaining a compact parameter budget.

Abstract

Prompt learning has demonstrated promising results in fine-tuning pre-trained multimodal models. However, the performance improvement is limited when applied to more complex and fine-grained tasks. The reason is that most existing methods directly optimize the parameters involved in the prompt generation process through loss backpropagation, which constrains the richness and specificity of the prompt representations. In this paper, we propose Diffusion-Driven Prompt Generator (Diff-Prompt), aiming to use the diffusion model to generate rich and fine-grained prompt information for complex downstream tasks. Specifically, our approach consists of three stages. In the first stage, we train a Mask-VAE to compress the masks into latent space. In the second stage, we leverage an improved Diffusion Transformer (DiT) to train a prompt generator in the latent space, using the masks for supervision. In the third stage, we align the denoising process of the prompt generator with the pre-trained model in the semantic space, and use the generated prompts to fine-tune the model. We conduct experiments on a complex pixel-level downstream task, referring expression comprehension, and compare our method with various parameter-efficient fine-tuning approaches. Diff-Prompt achieves a maximum improvement of 8.87 in R@1 and 14.05 in R@5 compared to the foundation model and also outperforms other state-of-the-art methods across multiple metrics. The experimental results validate the effectiveness of our approach and highlight the potential of using generative models for prompt generation. Code is available at https://github.com/Kelvin-ywc/diff-prompt.

Diff-Prompt: Diffusion-Driven Prompt Generator with Mask Supervision

TL;DR

Diff-Prompt tackles the limited gains of prompt learning for fine-grained multimodal tasks by using diffusion in latent space to generate rich, input-specific prompts. It introduces a three-stage pipeline: (1) Mask-VAE compresses masks into a latent representation, (2) a diffusion-based prompt generator creates prompts conditioned on image and caption, and (3) adapters align the generated prompts with the foundation model for fine-tuning, complemented by global prompts. Empirically, Diff-Prompt yields consistent improvements on RefCOCO and Flickr30k over various adapter/prompt baselines, demonstrates deeper prompt interactions enhancing performance, and shows promising cross-dataset/domain generalization and zero-shot capabilities. The approach underscores the potential of generative priors for prompt synthesis, enabling richer cross-modal guidance while maintaining a compact parameter budget.

Abstract

Prompt learning has demonstrated promising results in fine-tuning pre-trained multimodal models. However, the performance improvement is limited when applied to more complex and fine-grained tasks. The reason is that most existing methods directly optimize the parameters involved in the prompt generation process through loss backpropagation, which constrains the richness and specificity of the prompt representations. In this paper, we propose Diffusion-Driven Prompt Generator (Diff-Prompt), aiming to use the diffusion model to generate rich and fine-grained prompt information for complex downstream tasks. Specifically, our approach consists of three stages. In the first stage, we train a Mask-VAE to compress the masks into latent space. In the second stage, we leverage an improved Diffusion Transformer (DiT) to train a prompt generator in the latent space, using the masks for supervision. In the third stage, we align the denoising process of the prompt generator with the pre-trained model in the semantic space, and use the generated prompts to fine-tune the model. We conduct experiments on a complex pixel-level downstream task, referring expression comprehension, and compare our method with various parameter-efficient fine-tuning approaches. Diff-Prompt achieves a maximum improvement of 8.87 in R@1 and 14.05 in R@5 compared to the foundation model and also outperforms other state-of-the-art methods across multiple metrics. The experimental results validate the effectiveness of our approach and highlight the potential of using generative models for prompt generation. Code is available at https://github.com/Kelvin-ywc/diff-prompt.
Paper Structure (34 sections, 10 equations, 11 figures, 14 tables)

This paper contains 34 sections, 10 equations, 11 figures, 14 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Vision-language models
  4. Prompt Tuning
  5. Diffusion Models
  6. Preliminary
  7. Promblem Formulation and Foundation Model
  8. Deep Prompting
  9. Diffusion Models
  10. Diff-Prompt: Diffusion-driven Prompt Generator
  11. Mask-VAE Training
  12. Visual Prompt Generation using Diffusion Model
  13. Visual Prompt Tuning with Foundational Models
  14. Experiment
  15. Experiment Setup
  16. ...and 19 more sections

Figures (11)

  • Figure 1: (a) Comparison between mainstream prompt learning methods (the first two paradigms) and our Diff-prompt paradigm. (b) Comparison of different efficient fine-tuning methods on the RefCOCO dataset, with the x-axis representing R@1, the y-axis representing R@5, and bubble size indicating the total model parameters. Diff-Prompt achieves higher performance at the cost of using partial parameters.
  • Figure 2: The framework of Diffusion-Driven Prompt Generator (Diff-Prompt). We fully utilize a diffusion model as the prompt generator, which generates prompts conditioned on a given image and caption. The generated prompts are then mapped into input-specific prompts through modality-specific adapters. These input-specific prompts are concatenated with global prompts of equal length to form the final prompts, which are used to fine-tune the pre-trained model.
  • Figure 3: Forward and Sample Process of the Prompt Generator.
  • Figure 4: Qualitative Analysis for RefCOCO: Ground Truth (left), GLIP-T(A) (middle), Diff-Prompt (right). The results show the top three bounding boxes with the highest confidence, represented by green, blue, and purple from highest to lowest confidence, respectively. In the caption, the red content indicates positive tokens.
  • Figure 5: Metrics at Different Prompt Depths.
  • ...and 6 more figures