Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DialCLIP: Empowering CLIP as Multi-Modal Dialog Retriever

Zhichao Yin, Binyuan Hui, Min Yang, Fei Huang, Yongbin Li

TL;DR

DialCLIP tackles the challenge of using CLIP for multi-modal dialog retrieval by injecting dialog context through a context prompt generator, aligning multi-modal context via a domain prompt, and using a Mixture of Projection to support multiple retrieval types. The method achieves state-of-the-art results on PhotoChat and MMDialog with only 0.04% of parameters updated. The key technical contributions are the context prompt generator, domain prompts via deep prompt tuning, and the mixture of projection that handles different modalities and retrieval types, trained with a contrastive loss on multimodal pairs. The results demonstrate that carefully designed prompts can impart dialog-awareness to fixed vision-language models, enabling efficient, scalable dialog retrieval.

Abstract

Recently, substantial advancements in pre-trained vision-language models have greatly enhanced the capabilities of multi-modal dialog systems. These models have demonstrated significant improvements by fine-tuning on downstream tasks. However, the existing pre-trained models primarily focus on effectively capturing the alignment between vision and language modalities, often ignoring the intricate nature of dialog context. In this paper, we propose a parameter-efficient prompt-tuning method named DialCLIP for multi-modal dialog retrieval. Specifically, our approach introduces a multi-modal context prompt generator to learn context features which are subsequently distilled into prompts within the pre-trained vision-language model CLIP. Besides, we introduce domain prompt to mitigate the disc repancy from the downstream dialog data. To facilitate various types of retrieval, we also design multiple experts to learn mappings from CLIP outputs to multi-modal representation space, with each expert being responsible to one specific retrieval type. Extensive experiments show that DialCLIP achieves state-of-the-art performance on two widely recognized benchmark datasets (i.e., PhotoChat and MMDialog) by tuning a mere 0.04% of the total parameters. These results highlight the efficacy and efficiency of our proposed approach, underscoring its potential to advance the field of multi-modal dialog retrieval.

DialCLIP: Empowering CLIP as Multi-Modal Dialog Retriever

TL;DR

DialCLIP tackles the challenge of using CLIP for multi-modal dialog retrieval by injecting dialog context through a context prompt generator, aligning multi-modal context via a domain prompt, and using a Mixture of Projection to support multiple retrieval types. The method achieves state-of-the-art results on PhotoChat and MMDialog with only 0.04% of parameters updated. The key technical contributions are the context prompt generator, domain prompts via deep prompt tuning, and the mixture of projection that handles different modalities and retrieval types, trained with a contrastive loss on multimodal pairs. The results demonstrate that carefully designed prompts can impart dialog-awareness to fixed vision-language models, enabling efficient, scalable dialog retrieval.

Abstract

Recently, substantial advancements in pre-trained vision-language models have greatly enhanced the capabilities of multi-modal dialog systems. These models have demonstrated significant improvements by fine-tuning on downstream tasks. However, the existing pre-trained models primarily focus on effectively capturing the alignment between vision and language modalities, often ignoring the intricate nature of dialog context. In this paper, we propose a parameter-efficient prompt-tuning method named DialCLIP for multi-modal dialog retrieval. Specifically, our approach introduces a multi-modal context prompt generator to learn context features which are subsequently distilled into prompts within the pre-trained vision-language model CLIP. Besides, we introduce domain prompt to mitigate the disc repancy from the downstream dialog data. To facilitate various types of retrieval, we also design multiple experts to learn mappings from CLIP outputs to multi-modal representation space, with each expert being responsible to one specific retrieval type. Extensive experiments show that DialCLIP achieves state-of-the-art performance on two widely recognized benchmark datasets (i.e., PhotoChat and MMDialog) by tuning a mere 0.04% of the total parameters. These results highlight the efficacy and efficiency of our proposed approach, underscoring its potential to advance the field of multi-modal dialog retrieval.
Paper Structure (10 sections, 4 equations, 2 figures, 6 tables)

This paper contains 10 sections, 4 equations, 2 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Task Definition
  4. Model Architecture
  5. Model Training
  6. Experiments
  7. Experimental Setup
  8. Main Results
  9. Analysis
  10. Conclusion

Figures (2)

  • Figure 1: Example of retrieve-based multi-modal dialog system.
  • Figure 2: The architecture of our multi-modal response retrieval model (called DialCLIP), which consists of context prompt generator (CPG), CLIP encoders, and mixture of projection (MoP).