Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GDCNet: Generative Discrepancy Comparison Network for Multimodal Sarcasm Detection

Shuguang Zhang, Junhong Lian, Guoxin Yu, Baoxun Xu, Xiang Ao

TL;DR

GDCNet targets multimodal sarcasm detection by decoupling image caption generation from textual input and using factually grounded captions as stable semantic anchors. The Generative Discrepancy Representation Module jointly analyzes semantic, sentiment, and visual-text fidelity discrepancies between the generated description and the original text, and these cues are integrated with visual and textual features via a gated fusion mechanism. The approach achieves a new state-of-the-art on MMSD2.0, showing improved accuracy and robustness over LLM-based baselines and prior multimodal models. This work demonstrates that objective cross-modal anchors combined with adaptive discrepancy modeling can effectively capture subtle cross-modal incongruities in sarcasm detection, with potential applicability to other multimodal understanding tasks.

Abstract

Multimodal sarcasm detection (MSD) aims to identify sarcasm within image-text pairs by modeling semantic incongruities across modalities. Existing methods often exploit cross-modal embedding misalignment to detect inconsistency but struggle when visual and textual content are loosely related or semantically indirect. While recent approaches leverage large language models (LLMs) to generate sarcastic cues, the inherent diversity and subjectivity of these generations often introduce noise. To address these limitations, we propose the Generative Discrepancy Comparison Network (GDCNet). This framework captures cross-modal conflicts by utilizing descriptive, factually grounded image captions generated by Multimodal LLMs (MLLMs) as stable semantic anchors. Specifically, GDCNet computes semantic and sentiment discrepancies between the generated objective description and the original text, alongside measuring visual-textual fidelity. These discrepancy features are then fused with visual and textual representations via a gated module to adaptively balance modality contributions. Extensive experiments on MSD benchmarks demonstrate GDCNet's superior accuracy and robustness, establishing a new state-of-the-art on the MMSD2.0 benchmark.

GDCNet: Generative Discrepancy Comparison Network for Multimodal Sarcasm Detection

TL;DR

GDCNet targets multimodal sarcasm detection by decoupling image caption generation from textual input and using factually grounded captions as stable semantic anchors. The Generative Discrepancy Representation Module jointly analyzes semantic, sentiment, and visual-text fidelity discrepancies between the generated description and the original text, and these cues are integrated with visual and textual features via a gated fusion mechanism. The approach achieves a new state-of-the-art on MMSD2.0, showing improved accuracy and robustness over LLM-based baselines and prior multimodal models. This work demonstrates that objective cross-modal anchors combined with adaptive discrepancy modeling can effectively capture subtle cross-modal incongruities in sarcasm detection, with potential applicability to other multimodal understanding tasks.

Abstract

Multimodal sarcasm detection (MSD) aims to identify sarcasm within image-text pairs by modeling semantic incongruities across modalities. Existing methods often exploit cross-modal embedding misalignment to detect inconsistency but struggle when visual and textual content are loosely related or semantically indirect. While recent approaches leverage large language models (LLMs) to generate sarcastic cues, the inherent diversity and subjectivity of these generations often introduce noise. To address these limitations, we propose the Generative Discrepancy Comparison Network (GDCNet). This framework captures cross-modal conflicts by utilizing descriptive, factually grounded image captions generated by Multimodal LLMs (MLLMs) as stable semantic anchors. Specifically, GDCNet computes semantic and sentiment discrepancies between the generated objective description and the original text, alongside measuring visual-textual fidelity. These discrepancy features are then fused with visual and textual representations via a gated module to adaptively balance modality contributions. Extensive experiments on MSD benchmarks demonstrate GDCNet's superior accuracy and robustness, establishing a new state-of-the-art on the MMSD2.0 benchmark.
Paper Structure (14 sections, 7 equations, 2 figures, 4 tables)

This paper contains 14 sections, 7 equations, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Problem Formulation
  4. Cross-modal Feature Alignment
  5. Generative Discrepancy Representation Module
  6. Gated Multimodal Fusion & Classification
  7. Optimization Objective
  8. Experiments
  9. Experiments Details
  10. Main Results
  11. Ablation Study
  12. Comparison with LLM-based Methods
  13. MLLM Contribution Analysis
  14. Conclusion

Figures (2)

  • Figure 1: LLMs outputs on the same image: sarcastic explanations diverge across models and prompts, whereas factual descriptions remain stable, highlighting their potential as reliable semantic anchors.
  • Figure 2: The Architecture of GDCNet. The Gated Multimodal Fusion & Classification module integrates discrepancy ($F_D$), text ($F_T$), and image ($F_I$) features to produce the fused representation ($F_{\text{fused}}$).