Evaluating Video Models as Simulators of Multi-Person Pedestrian Trajectories

TL;DR

This work introduces a principled protocol to benchmark text-to-video and image-to-video models as implicit simulators of multi-person pedestrian dynamics. By extracting metric-scale trajectories from generated videos using BEV reconstructions (with known homographies for I2V and SfM/metric depth for T2V) and a 12-metric evaluation suite spanning trajectory realism, social interaction, and video fidelity, the authors quantify how well current models capture crowd dynamics. The results show that leading models learn useful priors for plausible multi-agent behavior and respond to prompts with varying densities and interactions, but persistent failure modes such as merging and disappearing pedestrians limit physical plausibility. The work provides a foundation and public dataset for advancing world models toward reliable long-range, multi-agent crowd simulations with explicit evaluation benchmarks.

Abstract

Large-scale video generation models have demonstrated high visual realism in diverse contexts, spurring interest in their potential as general-purpose world simulators. Existing benchmarks focus on individual subjects rather than scenes with multiple interacting people. However, the plausibility of multi-agent dynamics in generated videos remains unverified. We propose a rigorous evaluation protocol to benchmark text-to-video (T2V) and image-to-video (I2V) models as implicit simulators of pedestrian dynamics. For I2V, we leverage start frames from established datasets to enable comparison with a ground truth video dataset. For T2V, we develop a prompt suite to explore diverse pedestrian densities and interactions. A key component is a method to reconstruct 2D bird's-eye view trajectories from pixel-space without known camera parameters. Our analysis reveals that leading models have learned surprisingly effective priors for plausible multi-agent behavior. However, failure modes like merging and disappearing people highlight areas for future improvement.

Figures (13)

• Figure 1: From a text prompt, a video model generates a scene featuring pedestrian dynamics. We extract metric-scale trajectories from the synthetic video using 3D reconstruction to recover scene geometry and camera parameters, multi-object tracking to identify pedestrian paths in pixel-space, and projection of these paths into a unified bird's-eye view (BEV) coordinate system. The resulting trajectories are then analyzed for dynamic realism.
• Figure 2: We create prompt suites for T2V and I2V generation, a large-scale dataset of generated videos with extracted trajectories, and a comprehensive evaluation protocol assessing trajectory kinematics, social interaction, and video fidelity (data and code to be released).
• Figure 3: (a) T2V trajectory extraction via 3D reconstruction, versus (b) I2V comparison against ground truth via known homography.
• Figure 4: A 5-second excerpt from the UNIV scene of the ETH/UCY benchmark showing the ground truth (top row) and sample video generations using first-frame conditioning. Green borders indicate conditioning start frames.
• Figure 5: T2V results highlight the models' ability to generate complex social behaviors and scenes from text prompts. Additional larger visualizations provided in Appendix Figure \ref{['fig:qualitative_examples']}.