R^3-VQA: "Read the Room" by Video Social Reasoning

TL;DR

R^3-VQA introduces a high-fidelity video dataset for social reasoning that jointly encodes social events, mental states, and causal chains, addressing gaps in existing benchmarks that lack fine-grained multimodal cues. The dataset comprises 316 videos with 347 social causal chains, 2198 nodes, and 4840 generated plus 316 human-designed QA pairs, organized into event understanding, mental-state estimation, and causal questions. Experiments across diverse LVLMs show that current models lag behind humans in both accuracy and especially in consistency of reasoning, though ToM-inspired prompting and subtitles improve performance. The work highlights the challenges of reasoning about beliefs, desires, emotions, and causal relationships in real-world videos, suggesting ToM-enhanced multimodal prompting as a promising but still insufficient bridge to human-level social intelligence.

Abstract

"Read the room" is a significant social reasoning capability in human daily life. Humans can infer others' mental states from subtle social cues. Previous social reasoning tasks and datasets lack complexity (e.g., simple scenes, basic interactions, incomplete mental state variables, single-step reasoning, etc.) and fall far short of the challenges present in real-life social interactions. In this paper, we contribute a valuable, high-quality, and comprehensive video dataset named R^3-VQA with precise and fine-grained annotations of social events and mental states (i.e., belief, intent, desire, and emotion) as well as corresponding social causal chains in complex social scenarios. Moreover, we include human-annotated and model-generated QAs. Our task R^3-VQA includes three aspects: Social Event Understanding, Mental State Estimation, and Social Causal Reasoning. As a benchmark, we comprehensively evaluate the social reasoning capabilities and consistencies of current state-of-the-art large vision-language models (LVLMs). Comprehensive experiments show that (i) LVLMs are still far from human-level consistent social reasoning in complex social scenarios; (ii) Theory of Mind (ToM) prompting can help LVLMs perform better on social reasoning tasks. We provide some of our dataset and codes in supplementary material and will release our full dataset and codes upon acceptance.

Figures (6)

• Figure 1: The visible physical world we live in is just the tip of the iceberg compared to the vast, invisible mental world behind it zhu2020dark. In this exampleyoutube_video, we can observe that a brief moment of social interaction involves a series of complex and dynamic mental activities: B extends his hand to shake with A, but A fails to notice. B then pretends that his outstretched hand was meant to touch his head, attempting to conceal his embarrassment. Despite this, C sees through B's mental state and pats him on the shoulder to offer comfort. In response, B shrugs and gestures self-deprecatingly to ease the awkwardness. Social reasoning is a critical aspect of social intelligence. However, in long-term, highly random and dynamic social interactions, capturing subtle cues, recognizing social events, accurately estimating various mental states, and identifying complex reasoning chains become progressively challenging, making social reasoning even more intricate.
• Figure 2: We generally illustrate our dataset design (see \ref{['subsec: dataset design']}).
• Figure 3: Examples of each QA type. The option marked in green is the correct answer.
• Figure 4: Our dataset construction pipeline, which consists of five stages.
• Figure 5: Statistics of generated QA data.