SVBench: Evaluation of Video Generation Models on Social Reasoning

TL;DR

SVBench targets the core problem that state-of-the-art text-to-video models produce visually realistic scenes but struggle to generate socially coherent behavior. It introduces a training-free, four-agent pipeline that converts thirty psychology experiments into short-video prompts and uses a vision–language judge to assess five social-reasoning dimensions, quantified by $S_{\mathrm{overall}} = \frac{1}{5} \sum_{k=1}^{5} D_k$. Across eight contemporary video generators, SVBench reveals substantial gaps in intention understanding, belief reasoning, joint attention, and prosocial inference, despite strong surface realism in top models. The benchmark provides a scalable, theoretically grounded framework for evaluating social reasoning in video generation, enabling more robust, human-aligned AI systems with broader real-world impact.

Abstract

Recent text-to-video generation models exhibit remarkable progress in visual realism, motion fidelity, and text-video alignment, yet they remain fundamentally limited in their ability to generate socially coherent behavior. Unlike humans, who effortlessly infer intentions, beliefs, emotions, and social norms from brief visual cues, current models tend to render literal scenes without capturing the underlying causal or psychological logic. To systematically evaluate this gap, we introduce the first benchmark for social reasoning in video generation. Grounded in findings from developmental and social psychology, our benchmark organizes thirty classic social cognition paradigms into seven core dimensions, including mental-state inference, goal-directed action, joint attention, social coordination, prosocial behavior, social norms, and multi-agent strategy. To operationalize these paradigms, we develop a fully training-free agent-based pipeline that (i) distills the reasoning mechanism of each experiment, (ii) synthesizes diverse video-ready scenarios, (iii) enforces conceptual neutrality and difficulty control through cue-based critique, and (iv) evaluates generated videos using a high-capacity VLM judge across five interpretable dimensions of social reasoning. Using this framework, we conduct the first large-scale study across seven state-of-the-art video generation systems. Our results reveal substantial performance gaps: while modern models excel in surface-level plausibility, they systematically fail in intention recognition, belief reasoning, joint attention, and prosocial inference.

Figures (4)

• Figure 1: (a) Social reasoning scenario. (b) Our benchmark framework, which uses a two-part agent-based pipeline for constructing and evaluating social reasoning tasks in video generation.
• Figure 2: Pipeline overview. The framework consists of two training-free components: (1) an agent-based generation pipeline that transforms psychologically grounded social reasoning experiments into diverse, difficulty-controlled video prompts, and (2) an agent-based evaluation pipeline that uses a vision–language model to score generated videos along five discrete dimensions of social reasoning.
• Figure 3: Comparison between automated and human evaluation. (a) Scores from the agent-based VLM Judge. (b) Scores from human annotators using the same five-dimensional rubric. The two profiles exhibit closely matched trends across dimensions and models.
• Figure 4: Representative qualitative cases of the agent-based evaluator. Each panel shows sampled video frames, the generation prompt, the five-dimensional scores, and the model’s natural-language justification.