Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Can Image-To-Video Models Simulate Pedestrian Dynamics?

Aaron Appelle, Jerome P. Lynch

TL;DR

The paper investigates whether image-to-video diffusion-transformer models can simulate pedestrian dynamics in crowded scenes by conditioning video generation on ETH/UCY keyframes and evaluating trajectories with a FairMOT-based pipeline. It introduces a benchmarking framework that generates synthetic sequences from three state-of-the-art I2V models, then extracts world-coordinate trajectories and computes classic crowd metrics. Results show that while these models capture general motion and environment interactions, they fall short on multi-agent interactions such as passing distance, with no model matching ground-truth across all scenarios. The study highlights both the potential and current limitations of I2V models as world simulators for pedestrian dynamics, and suggests scaling to larger models or alternative training data to improve social-behavior fidelity.

Abstract

Recent high-performing image-to-video (I2V) models based on variants of the diffusion transformer (DiT) have displayed remarkable inherent world-modeling capabilities by virtue of training on large scale video datasets. We investigate whether these models can generate realistic pedestrian movement patterns in crowded public scenes. Our framework conditions I2V models on keyframes extracted from pedestrian trajectory benchmarks, then evaluates their trajectory prediction performance using quantitative measures of pedestrian dynamics.

Can Image-To-Video Models Simulate Pedestrian Dynamics?

TL;DR

The paper investigates whether image-to-video diffusion-transformer models can simulate pedestrian dynamics in crowded scenes by conditioning video generation on ETH/UCY keyframes and evaluating trajectories with a FairMOT-based pipeline. It introduces a benchmarking framework that generates synthetic sequences from three state-of-the-art I2V models, then extracts world-coordinate trajectories and computes classic crowd metrics. Results show that while these models capture general motion and environment interactions, they fall short on multi-agent interactions such as passing distance, with no model matching ground-truth across all scenarios. The study highlights both the potential and current limitations of I2V models as world simulators for pedestrian dynamics, and suggests scaling to larger models or alternative training data to improve social-behavior fidelity.

Abstract

Recent high-performing image-to-video (I2V) models based on variants of the diffusion transformer (DiT) have displayed remarkable inherent world-modeling capabilities by virtue of training on large scale video datasets. We investigate whether these models can generate realistic pedestrian movement patterns in crowded public scenes. Our framework conditions I2V models on keyframes extracted from pedestrian trajectory benchmarks, then evaluates their trajectory prediction performance using quantitative measures of pedestrian dynamics.
Paper Structure (6 sections, 2 figures, 1 table)

This paper contains 6 sections, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Method
  3. I2V Inference
  4. Postprocessing
  5. Results
  6. Conclusion

Figures (2)

  • Figure 1: A 5-second excerpt from the UNIV scene of the ETH/UCY benchmark showing sample video generations using first-frame conditioning. Green borders indicate conditioning frames, and all frames show tracklets of detected pedestrians. The extracted frames are equally spaced at 1.25 seconds.
  • Figure 2: Pedestrian dynamics evaluated on the UNIV scene. (a) position heatmaps; (b) velocity distribution with gaussian fit; (c) fundamental diagram; and (d) nearest neighbor polar heatmap.