Mind the Motions: Benchmarking Theory-of-Mind in Everyday Body Language

TL;DR

Motion2Mind presents a three-stage, dictionary-grounded framework for nonverbal Theory of Mind (ToM) evaluation, decomposing tasks into Detection, Knowledge, and Explanation. The authors build a realistic multimodal benchmark with a curated body-language dictionary (407 cues; 2,050 explanations) linked to 397 mind states, drawn from ~$497$ hours of YouTube video and short 4-second clips, annotated with fine-grained visual and vocal cues and contextual grounding. Across evaluation, current vision-language models exhibit a pronounced human–AI gap; larger models and ToM-enabled modules yield modest gains, with detection and explanation tasks being particularly challenging and over-interpretation common when cues are invalid. The dataset and protocol reveal practical implications for safe, reliable social AI and provide a robust platform for future ToM research, while acknowledging limitations related to cultural variability and potential misuse.

Abstract

Our ability to interpret others' mental states through nonverbal cues (NVCs) is fundamental to our survival and social cohesion. While existing Theory of Mind (ToM) benchmarks have primarily focused on false-belief tasks and reasoning with asymmetric information, they overlook other mental states beyond belief and the rich tapestry of human nonverbal communication. We present Motion2Mind, a framework for evaluating the ToM capabilities of machines in interpreting NVCs. Leveraging an expert-curated body-language reference as a proxy knowledge base, we build Motion2Mind, a carefully curated video dataset with fine-grained nonverbal cue annotations paired with manually verified psychological interpretations. It encompasses 222 types of nonverbal cues and 397 mind states. Our evaluation reveals that current AI systems struggle significantly with NVC interpretation, exhibiting not only a substantial performance gap in Detection, as well as patterns of over-interpretation in Explanation compared to human annotators.

Figures (8)

• Figure 1: We disentangle concept of nonverbal cue understanding into three distinct components: (1) Detection, identifying and labeling various naturalistic movements; (2) Knowledge, the general understanding of the psychological meanings associated with specific cues; and (3) Explanation, contextual reasoning to infer the psychological state behind observed cues. Our test set, developed based on Joe Navarro’s work, reveals that while LLMs perform comparably to humans in Knowledge, they exhibit a substantial gap in the Explanation and Detection phase.
• Figure 2: NVC knowledge scores of intelligent LLMs --- GPT (green), Claude (orange), Qwen2.5-Instruct (purple) --- tested on the NVC dictionary. LLMs manifest structurized knowledge even than psychological experts.
• Figure 3: We build Motion2Mind, a dataset annotated with fine-grained multimodal (m.m.) cues. To construct the dataset, we collect 497 hours of video from YouTube (sitcoms, movies, reality shows), sample short clips (32 frames), and generate initial captions using Qwen2.5-32B-VL-Instruct. These captions are filtered using a body-language dictionary to prioritize clips with interpretable cues and meanings. Human annotators then manually inspect the clips and refine the explanations based on contextual grounding, ensuring that each cue is paired with its most accurate and salient psychological meaning within the scene.
• Figure 4: Stacked bar plots of Explanation task answers. Small models shows low precision (over-interpret) compared to larger models.
• Figure 5: Examples of erroneous inferences by the GPT-O1 model in Detection-Binary and explanation tasks. The first row illustrates the example which model doesn't recognize the given cue (e.g. Smile, Neck touching). The second row presents misinterpretations, where benign or contextually ambiguous cues are incorrectly assigned psychological meanings (F: False explanation, T: True explanation).