Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mixture-of-Prompt-Experts for Multi-modal Semantic Understanding

Zichen Wu, Hsiu-Yuan Huang, Fanyi Qu, Yunfang Wu

TL;DR

This paper proposes Mixture-of-Prompt-Experts with Block-Aware Prompt Fusion (MoPE-BAF), a novel multi-modal soft prompt framework based on the unified vision-language model (VLM), which significantly outperforms other widely-used prompt methods on VLMs or task-specific methods.

Abstract

Deep multimodal semantic understanding that goes beyond the mere superficial content relation mining has received increasing attention in the realm of artificial intelligence. The challenges of collecting and annotating high-quality multi-modal data have underscored the significance of few-shot learning. In this paper, we focus on two critical tasks under this context: few-shot multi-modal sarcasm detection (MSD) and multi-modal sentiment analysis (MSA). To address them, we propose Mixture-of-Prompt-Experts with Block-Aware Prompt Fusion (MoPE-BAF), a novel multi-modal soft prompt framework based on the unified vision-language model (VLM). Specifically, we design three experts of soft prompts: a text prompt and an image prompt that extract modality-specific features to enrich the single-modal representation, and a unified prompt to assist multi-modal interaction. Additionally, we reorganize Transformer layers into several blocks and introduce cross-modal prompt attention between adjacent blocks, which smoothens the transition from single-modal representation to multi-modal fusion. On both MSD and MSA datasets in few-shot setting, our proposed model not only surpasses the 8.2B model InstructBLIP with merely 2% parameters (150M), but also significantly outperforms other widely-used prompt methods on VLMs or task-specific methods.

Mixture-of-Prompt-Experts for Multi-modal Semantic Understanding

TL;DR

This paper proposes Mixture-of-Prompt-Experts with Block-Aware Prompt Fusion (MoPE-BAF), a novel multi-modal soft prompt framework based on the unified vision-language model (VLM), which significantly outperforms other widely-used prompt methods on VLMs or task-specific methods.

Abstract

Deep multimodal semantic understanding that goes beyond the mere superficial content relation mining has received increasing attention in the realm of artificial intelligence. The challenges of collecting and annotating high-quality multi-modal data have underscored the significance of few-shot learning. In this paper, we focus on two critical tasks under this context: few-shot multi-modal sarcasm detection (MSD) and multi-modal sentiment analysis (MSA). To address them, we propose Mixture-of-Prompt-Experts with Block-Aware Prompt Fusion (MoPE-BAF), a novel multi-modal soft prompt framework based on the unified vision-language model (VLM). Specifically, we design three experts of soft prompts: a text prompt and an image prompt that extract modality-specific features to enrich the single-modal representation, and a unified prompt to assist multi-modal interaction. Additionally, we reorganize Transformer layers into several blocks and introduce cross-modal prompt attention between adjacent blocks, which smoothens the transition from single-modal representation to multi-modal fusion. On both MSD and MSA datasets in few-shot setting, our proposed model not only surpasses the 8.2B model InstructBLIP with merely 2% parameters (150M), but also significantly outperforms other widely-used prompt methods on VLMs or task-specific methods.
Paper Structure (32 sections, 8 equations, 4 figures, 6 tables)

This paper contains 32 sections, 8 equations, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multi-modal Sarcasm Detection
  4. Multi-modal Sentiment Analysis
  5. Multi-modal Prompt Learning
  6. Preliminary
  7. The Vision-language Pre-trained Model: VLMo
  8. Tuning on Multi-modal Tasks
  9. Finetuning.
  10. Manual Prompt.
  11. Soft Prompt.
  12. The Proposed Model
  13. Mixture-of-Prompt-Experts
  14. Stage 1.
  15. Stage 2.
  16. ...and 17 more sections

Figures (4)

  • Figure 1: The framework of VLMo for image-text detection task. We demonstrate two methods. In finetuning, the [CLS] representation is fed to a classification head, while in manual prompt, the representation of [MASK] is fed to a verbalizer.
  • Figure 2: Our proposed MoPE-BAF model for multi-modal semantic understanding .
  • Figure 3: Receptive fields of different prompts, image patches, text tokens in the self-attention module when using MoPE. VP and LP are shorthand for V-Prompt, L-Prompt.
  • Figure 4: (a) F1 performance training MoPE with different prompt lengths. (b) F1 scores training MoPE-BAF with different block numbers. (c) Comparison between VLMo and VLMo + MoPE-BAF under different training shots.