MUStReason: A Benchmark for Diagnosing Pragmatic Reasoning in Video-LMs for Multimodal Sarcasm Detection

TL;DR

This work addresses the challenge of pragmatic reasoning in multimodal sarcasm detection by introducing MUStReason, a diagnostic benchmark with fine-grained modality cue annotations and reasoning steps. It pairs MUStReason with PragCoT, a structured prompting framework that separates perception and reasoning to focus on implied intentions rather than literal content. Through extensive experiments across VideoLMs, the authors demonstrate that PragCoT improves sarcasm classification and provide analyses distinguishing perceptual versus reasoning failures, highlighting current gaps in model capabilities. The benchmark and framework together offer a pathway to more interpretable, pragmatics-aware multimodal systems with potential extensions to humor and figurative language understanding.

Abstract

Sarcasm is a specific type of irony which involves discerning what is said from what is meant. Detecting sarcasm depends not only on the literal content of an utterance but also on non-verbal cues such as speaker's tonality, facial expressions and conversational context. However, current multimodal models struggle with complex tasks like sarcasm detection, which require identifying relevant cues across modalities and pragmatically reasoning over them to infer the speaker's intention. To explore these limitations in VideoLMs, we introduce MUStReason, a diagnostic benchmark enriched with annotations of modality-specific relevant cues and underlying reasoning steps to identify sarcastic intent. In addition to benchmarking sarcasm classification performance in VideoLMs, using MUStReason we quantitatively and qualitatively evaluate the generated reasoning by disentangling the problem into perception and reasoning, we propose PragCoT, a framework that steers VideoLMs to focus on implied intentions over literal meaning, a property core to detecting sarcasm.

Figures (5)

• Figure 1: Sarcasm Reasoning Generation Pipeline for MUStReason
• Figure 2: Comparison of prompting strategies for multimodal sarcasm reasoning. PragCoT extends standard CoT by explicitly decoding perceptual cues before reasoning and classification.
• Figure 3: Human Evaluation of Annotation Quality in MUStReason.
• Figure 4: Qualitative Evaluation of Model Generated Reasoning.
• Figure 5: Qualitative and quantitative analysis of model-generated reasoning using annotations from the MUStReason dataset. These annotations help in identifying whether the model fails in perception, reasoning or both when identifying sarcasm. Bold words in Gold indicate the attributes present in Predicted. Text marked in green and red indicate correct and wrong predictions respectively. Tags $<$P$>$ and $<$R$>$ represent perception and reasoning respectively.