Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Can MLLMs Read the Room? A Multimodal Benchmark for Assessing Deception in Multi-Party Social Interactions

Caixin Kang, Yifei Huang, Liangyang Ouyang, Mingfang Zhang, Ruicong Liu, Yoichi Sato

TL;DR

The paper introduces MIDA, a challenging multimodal benchmark for deception reasoning in multi-party social interactions, operationalized through the Werewolf game to provide verifiable ground truth for every statement. It demonstrates that state-of-the-art MLLMs struggle to ground language in multimodal social cues and to model others' knowledge and beliefs, exposing a gap in Theory of Mind capabilities. To address this, the authors propose SoCoT, a Social Chain-of-Thought pipeline, and DSEM, a Dynamic Social Epistemic Memory module, which yield measurable improvements across a diverse set of models. The work highlights critical gaps and outlines a concrete path toward more perceptive and trustworthy AI systems capable of nuanced social reasoning in complex interactions.

Abstract

Despite their advanced reasoning capabilities, state-of-the-art Multimodal Large Language Models (MLLMs) demonstrably lack a core component of human intelligence: the ability to `read the room' and assess deception in complex social interactions. To rigorously quantify this failure, we introduce a new task, Multimodal Interactive Deception Assessment (MIDA), and present a novel multimodal dataset providing synchronized video and text with verifiable ground-truth labels for every statement. We establish a comprehensive benchmark evaluating 12 state-of-the-art open- and closed-source MLLMs, revealing a significant performance gap: even powerful models like GPT-4o struggle to distinguish truth from falsehood reliably. Our analysis of failure modes indicates that these models fail to effectively ground language in multimodal social cues and lack the ability to model what others know, believe, or intend, highlighting the urgent need for novel approaches to building more perceptive and trustworthy AI systems. To take a step forward, we design a Social Chain-of-Thought (SoCoT) reasoning pipeline and a Dynamic Social Epistemic Memory (DSEM) module. Our framework yields performance improvement on this challenging task, demonstrating a promising new path toward building MLLMs capable of genuine human-like social reasoning.

Can MLLMs Read the Room? A Multimodal Benchmark for Assessing Deception in Multi-Party Social Interactions

TL;DR

The paper introduces MIDA, a challenging multimodal benchmark for deception reasoning in multi-party social interactions, operationalized through the Werewolf game to provide verifiable ground truth for every statement. It demonstrates that state-of-the-art MLLMs struggle to ground language in multimodal social cues and to model others' knowledge and beliefs, exposing a gap in Theory of Mind capabilities. To address this, the authors propose SoCoT, a Social Chain-of-Thought pipeline, and DSEM, a Dynamic Social Epistemic Memory module, which yield measurable improvements across a diverse set of models. The work highlights critical gaps and outlines a concrete path toward more perceptive and trustworthy AI systems capable of nuanced social reasoning in complex interactions.

Abstract

Despite their advanced reasoning capabilities, state-of-the-art Multimodal Large Language Models (MLLMs) demonstrably lack a core component of human intelligence: the ability to `read the room' and assess deception in complex social interactions. To rigorously quantify this failure, we introduce a new task, Multimodal Interactive Deception Assessment (MIDA), and present a novel multimodal dataset providing synchronized video and text with verifiable ground-truth labels for every statement. We establish a comprehensive benchmark evaluating 12 state-of-the-art open- and closed-source MLLMs, revealing a significant performance gap: even powerful models like GPT-4o struggle to distinguish truth from falsehood reliably. Our analysis of failure modes indicates that these models fail to effectively ground language in multimodal social cues and lack the ability to model what others know, believe, or intend, highlighting the urgent need for novel approaches to building more perceptive and trustworthy AI systems. To take a step forward, we design a Social Chain-of-Thought (SoCoT) reasoning pipeline and a Dynamic Social Epistemic Memory (DSEM) module. Our framework yields performance improvement on this challenging task, demonstrating a promising new path toward building MLLMs capable of genuine human-like social reasoning.

Paper Structure

This paper contains 19 sections, 1 equation, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multimodal Social Interaction
  4. Deception Detection
  5. Computational Modeling of Deduction Games
  6. The MIDA Benchmark
  7. Dataset Curation and Annotation
  8. Task Definition and Evaluation
  9. Dataset Analysis
  10. New Modules for Deception Reasoning
  11. SoCoT: Social Chain-of-Thought
  12. DSEM: Dynamic Social Epistemic Memory
  13. Experiments
  14. Experimental Setup
  15. Main Benchmark Results
  16. ...and 4 more sections

Figures (4)

  • Figure 1: Overview of the MIDA Benchmark and New Modules for Deception Reasoning. (A) A semi-automated process where game transcripts, rules, and metadata are used by an LLM assistant to generate veracity labels. (B) The evaluated leading MLLMs. (C) Proposed new modules: (C.1) SoCoT grounds inference in multimodal cues, while (C.2) DSEM tracks participants' social states. (D) The MIDA task requires models to perform reasoning from Persuasive Strategy Classification to Deception Assessment with a supporting justification.
  • Figure 2: Persuasive Strategy proportion and distribution of veracity labels across Persuasive Strategy categories in MIDA-Ego4D/Youtube.
  • Figure 3: Demonstration of SoCoT and DSEM Modules. (Left) The SoCoT pipeline extracts multimodal behavioral primitives from Face, Body, and Voice. (Right) The DSEM module maintains a structured 'Memory Board' tracking participants' evolving beliefs and social states. (Bottom) An example of SoCoT's and DSEM's reasoning steps.
  • Figure 4: Radar chart of MLLMs’ accuracy in the MIDA task across Persuasive Strategy categories in MIDA-Ego4D/Youtube.