Think Twice Before You Judge: Mixture of Dual Reasoning Experts for Multimodal Sarcasm Detection

TL;DR

MiDRE tackles multimodal sarcasm detection by fusing internal content-based reasoning with external, rationale-guided reasoning via a large vision-language model. It introduces two specialized experts, IRE and ERE, plus a trainable gating mechanism that adaptively balances their contributions across encoder layers, guided by three-stage CoT rationales. Empirical results on MMSD2.0 and MMSD show state-of-the-art performance and improved robustness, with analyses revealing when external rationales are beneficial and how gating provides interpretability. The work demonstrates that structured, sarcasm-oriented external knowledge can significantly enhance multimodal understanding beyond surface cues.

Abstract

Multimodal sarcasm detection has attracted growing interest due to the rise of multimedia posts on social media. Understanding sarcastic image-text posts often requires external contextual knowledge, such as cultural references or commonsense reasoning. However, existing models struggle to capture the deeper rationale behind sarcasm, relying mainly on shallow cues like image captions or object-attribute pairs from images. To address this, we propose \textbf{MiDRE} (\textbf{Mi}xture of \textbf{D}ual \textbf{R}easoning \textbf{E}xperts), which integrates an internal reasoning expert for detecting incongruities within the image-text pair and an external reasoning expert that utilizes structured rationales generated via Chain-of-Thought prompting to a Large Vision-Language Model. An adaptive gating mechanism dynamically weighs the two experts, selecting the most relevant reasoning path. Unlike prior methods that treat external knowledge as static input, MiDRE selectively adapts to when such knowledge is beneficial, mitigating the risks of hallucinated or irrelevant signals from large models. Experiments on two benchmark datasets show that MiDRE achieves superior performance over baselines. Various qualitative analyses highlight the crucial role of external rationales, revealing that even when they are occasionally noisy, they provide valuable cues that guide the model toward a better understanding of sarcasm.

Figures (10)

• Figure 1: Comparison between external knowledge used by existing approaches vs our approach.
• Figure 2: Rationale generation module of MiDRE.
• Figure 3: Architecture of the proposed MiDRE model.
• Figure 4: Zero-shot Performance of LVLMs on MMSD2.0 test samples.
• Figure 5: Unique test samples correctly classified by MiDRE(w/o IRE) vs MiDRE(w/o ERE).