DrivingGen: A Comprehensive Benchmark for Generative Video World Models in Autonomous Driving

TL;DR

This paper addresses the absence of a driving-focused benchmark for generative world models. It introduces DrivingGen, combining a diverse Open-Domain and an Ego-Conditioned dataset with a four-part metric suite (distribution, quality, temporal consistency, trajectory alignment) to evaluate both visual realism and motion plausibility. Benchmark results across 14 models reveal clear trade-offs: general models tend to sacrifice physics for visual fidelity, while driving-specific approaches often improve motion realism but lag in image quality, with trajectory alignment under ego-conditioning still limited. By providing a unified, open framework for scalable simulation, safe testing of corner cases, and data-driven planning, DrivingGen aims to accelerate the development of reliable, controllable, and deployable driving world models.

Abstract

Video generation models, as one form of world models, have emerged as one of the most exciting frontiers in AI, promising agents the ability to imagine the future by modeling the temporal evolution of complex scenes. In autonomous driving, this vision gives rise to driving world models: generative simulators that imagine ego and agent futures, enabling scalable simulation, safe testing of corner cases, and rich synthetic data generation. Yet, despite fast-growing research activity, the field lacks a rigorous benchmark to measure progress and guide priorities. Existing evaluations remain limited: generic video metrics overlook safety-critical imaging factors; trajectory plausibility is rarely quantified; temporal and agent-level consistency is neglected; and controllability with respect to ego conditioning is ignored. Moreover, current datasets fail to cover the diversity of conditions required for real-world deployment. To address these gaps, we present DrivingGen, the first comprehensive benchmark for generative driving world models. DrivingGen combines a diverse evaluation dataset curated from both driving datasets and internet-scale video sources, spanning varied weather, time of day, geographic regions, and complex maneuvers, with a suite of new metrics that jointly assess visual realism, trajectory plausibility, temporal coherence, and controllability. Benchmarking 14 state-of-the-art models reveals clear trade-offs: general models look better but break physics, while driving-specific ones capture motion realistically but lag in visual quality. DrivingGen offers a unified evaluation framework to foster reliable, controllable, and deployable driving world models, enabling scalable simulation, planning, and data-driven decision-making.

Figures (5)

• Figure 1: Overview of our DrivingGen benchmark. Video models take vision, and optional language/action as inputs to generate videos. The generated videos are then passed into our evaluation suite. Four comprehensive and novel sets of metrics for both videos and trajectories (distribution, quality, temporal consistency, and trajectory alignment) are introduced to evaluate world models.
• Figure 2: Dataset distribution and gallery in our benchmark (top to bottom).
• Figure 3: The statistics of our ego-condition track.
• Figure 4: The gallery of our ego-condition track.
• Figure 5: Human Validation of Our benchmark. Our metrics closely match human preferences. Trajectory-related metrics are less accurate in comparison to humans, likely due to noisy monocular SLAM and metric-depth recovery from generated videos with artifacts.