Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MMSD3.0: A Multi-Image Benchmark for Real-World Multimodal Sarcasm Detection

Haochen Zhao, Yuyao Kong, Yongxiu Xu, Gaopeng Gou, Hongbo Xu, Yubin Wang, Haoliang Zhang

TL;DR

MMSD3.0 addresses a real-world gap in multimodal sarcasm detection by introducing a multi-image benchmark (2–4 images per instance) and a cross-image reasoning model, CIRM, that jointly models inter-image relations and text–image alignment via a Dual-Stage Bridge and Relevance-Guided Fusion. The dataset combines Twitter and Amazon reviews with careful annotation, OCR retention, and emojis to capture rich multimodal signals, enabling robust evaluation beyond single-image setups. Empirical results show CIRM achieves state-of-the-art performance across MMSD, MMSD2.0, and MMSD3.0, with strong ablations validating the importance of each component, and with notable robustness in real-world versus AI-generated data. The work advances practical sarcasm detection by highlighting the necessity of multi-image reasoning, OCR-grounded cues, and order-aware fusion for reliable multimodal understanding in noisy, real-world content.

Abstract

Despite progress in multimodal sarcasm detection, existing datasets and methods predominantly focus on single-image scenarios, overlooking potential semantic and affective relations across multiple images. This leaves a gap in modeling cases where sarcasm is triggered by multi-image cues in real-world settings. To bridge this gap, we introduce MMSD3.0, a new benchmark composed entirely of multi-image samples curated from tweets and Amazon reviews. We further propose the Cross-Image Reasoning Model (CIRM), which performs targeted cross-image sequence modeling to capture latent inter-image connections. In addition, we introduce a relevance-guided, fine-grained cross-modal fusion mechanism based on text-image correspondence to reduce information loss during integration. We establish a comprehensive suite of strong and representative baselines and conduct extensive experiments, showing that MMSD3.0 is an effective and reliable benchmark that better reflects real-world conditions. Moreover, CIRM demonstrates state-of-the-art performance across MMSD, MMSD2.0 and MMSD3.0, validating its effectiveness in both single-image and multi-image scenarios.

MMSD3.0: A Multi-Image Benchmark for Real-World Multimodal Sarcasm Detection

TL;DR

MMSD3.0 addresses a real-world gap in multimodal sarcasm detection by introducing a multi-image benchmark (2–4 images per instance) and a cross-image reasoning model, CIRM, that jointly models inter-image relations and text–image alignment via a Dual-Stage Bridge and Relevance-Guided Fusion. The dataset combines Twitter and Amazon reviews with careful annotation, OCR retention, and emojis to capture rich multimodal signals, enabling robust evaluation beyond single-image setups. Empirical results show CIRM achieves state-of-the-art performance across MMSD, MMSD2.0, and MMSD3.0, with strong ablations validating the importance of each component, and with notable robustness in real-world versus AI-generated data. The work advances practical sarcasm detection by highlighting the necessity of multi-image reasoning, OCR-grounded cues, and order-aware fusion for reliable multimodal understanding in noisy, real-world content.

Abstract

Despite progress in multimodal sarcasm detection, existing datasets and methods predominantly focus on single-image scenarios, overlooking potential semantic and affective relations across multiple images. This leaves a gap in modeling cases where sarcasm is triggered by multi-image cues in real-world settings. To bridge this gap, we introduce MMSD3.0, a new benchmark composed entirely of multi-image samples curated from tweets and Amazon reviews. We further propose the Cross-Image Reasoning Model (CIRM), which performs targeted cross-image sequence modeling to capture latent inter-image connections. In addition, we introduce a relevance-guided, fine-grained cross-modal fusion mechanism based on text-image correspondence to reduce information loss during integration. We establish a comprehensive suite of strong and representative baselines and conduct extensive experiments, showing that MMSD3.0 is an effective and reliable benchmark that better reflects real-world conditions. Moreover, CIRM demonstrates state-of-the-art performance across MMSD, MMSD2.0 and MMSD3.0, validating its effectiveness in both single-image and multi-image scenarios.
Paper Structure (27 sections, 23 equations, 11 figures, 7 tables)

This paper contains 27 sections, 23 equations, 11 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multimodal Sarcasm Detection
  4. Multimodal Large Language Models
  5. Dataset Construction
  6. Data Collection
  7. Human Annotation
  8. Data Statistics
  9. Methodology
  10. Problem Formulation
  11. Data Encoding
  12. Positional Encoding & Masking
  13. Dual-Stage Bridge Module
  14. Relevance-Guided Fusion
  15. Classification
  16. ...and 12 more sections

Figures (11)

  • Figure 1: An example of sarcasm created through the contrast between images.
  • Figure 2: OCR and emoji coverage comparison of two datasets.
  • Figure 3: Sarcastic vs Non-sarcastic Text Length Distribution.
  • Figure 4: The overall architecture of the Cross-Image Reasoning Model
  • Figure 5: Attention visualization of some examples. Red regions in the images indicate high visual attention, green moderate, and blue low. Text attention is shown below each image pair, with darker red denoting higher attention.
  • ...and 6 more figures