GeoDrive: 3D Geometry-Informed Driving World Model with Precise Action Control

TL;DR

GeoDrive addresses robust 3D geometric consistency in driving world models by constructing a monocular 3D representation, projecting it along an ego-trajectory, and using dynamic editing and diffusion-based refinement to produce geometrically faithful future frames. It introduces a neural-geometric framework that enforces spatio-temporal coherence and explicit trajectory conditioning, including a MonST3R-based 3D reconstruction, dynamic 2D editing, and a dual-branch diffusion refinement with a lightweight condition encoder. Empirical results on NuScenes show a $42\%$ reduction in ego-trajectory errors over a Vista baseline, improved video quality (LPIPS, PSNR, SSIM, FID, FVD), and zero-shot generalization to Waymo, along with interactive scene editing and VLA planning capabilities. These contributions enable safer, more reliable planning and data-generation for autonomous driving in diverse scenarios.

Abstract

Recent advancements in world models have revolutionized dynamic environment simulation, allowing systems to foresee future states and assess potential actions. In autonomous driving, these capabilities help vehicles anticipate the behavior of other road users, perform risk-aware planning, accelerate training in simulation, and adapt to novel scenarios, thereby enhancing safety and reliability. Current approaches exhibit deficiencies in maintaining robust 3D geometric consistency or accumulating artifacts during occlusion handling, both critical for reliable safety assessment in autonomous navigation tasks. To address this, we introduce GeoDrive, which explicitly integrates robust 3D geometry conditions into driving world models to enhance spatial understanding and action controllability. Specifically, we first extract a 3D representation from the input frame and then obtain its 2D rendering based on the user-specified ego-car trajectory. To enable dynamic modeling, we propose a dynamic editing module during training to enhance the renderings by editing the positions of the vehicles. Extensive experiments demonstrate that our method significantly outperforms existing models in both action accuracy and 3D spatial awareness, leading to more realistic, adaptable, and reliable scene modeling for safer autonomous driving. Additionally, our model can generalize to novel trajectories and offers interactive scene editing capabilities, such as object editing and object trajectory control.

Figures (10)

• Figure 1: GeoDrive enables precise trajectory following, correct novel view synthesis, and dynamic scene editing in autonomous driving scenarios. Our method integrates robust 3D conditions into driving world models, enhancing spatial understanding and action controllability.
• Figure 2: Overview of our training pipeline. We use a pretrained dense stereo model to obtain 3D point clouds and camera trajectories. A dynamic video is rendered from the first-frame point cloud using our dynamic editing technique. The noisy latent representation and rendered video are encoded via a VAE and concatenated as input for our condition encoder, modulating the DiT model's features. The DiT then generates photorealistic video that accurately follows the specified action conditions.
• Figure 3: Illustration of dynamic edit design. Compared with default rendering, it effectively reduces disparity between static rendering and dynamic real-world scenarios.
• Figure 4: Qualitative comparison of action fidelity under the same conditional frame and action control. Our model precisely follows desired trajectory, while Vista gao2024vista produce misaligned results.
• Figure 5: Qualitative Comparisons: Left - Enhanced visual fidelity in our predictions; Right - Superior scene dynamics understanding.