InstaDrive: Instance-Aware Driving World Models for Realistic and Consistent Video Generation

TL;DR

InstaDrive addresses two key shortcomings of driving world models—instance-level temporal consistency and spatial geometric fidelity—by introducing Instance Flow Guider (IFG) to propagate per-instance features across frames and Spatial Geometric Aligner (SGA) to align 3D bounding boxes with image-space constraints and occlusion order. The framework integrates ControlNet-informed denoising within a ST-DiT backbone, leveraging BEV-to-FPV projections and Fourier-embedded depth cues to enforce accurate spatial localization and depth-based occlusion. Evaluations on nuScenes show state-of-the-art video realism (e.g., $\text{FVD}=38.06$, $\text{FID}=3.96$) and strong performance on perception and multi-object tracking tasks, while CARLA-based procedurally generated long-tail scenarios demonstrate the approach’s utility for safety evaluation and scenario diversity. The work presents a practical pathway for high-fidelity, instance-aware synthetic driving data that supports downstream perception, tracking, and planning, along with rich long-tail scenario generation for autonomous systems.

Abstract

Autonomous driving relies on robust models trained on high-quality, large-scale multi-view driving videos. While world models offer a cost-effective solution for generating realistic driving videos, they struggle to maintain instance-level temporal consistency and spatial geometric fidelity. To address these challenges, we propose InstaDrive, a novel framework that enhances driving video realism through two key advancements: (1) Instance Flow Guider, which extracts and propagates instance features across frames to enforce temporal consistency, preserving instance identity over time. (2) Spatial Geometric Aligner, which improves spatial reasoning, ensures precise instance positioning, and explicitly models occlusion hierarchies. By incorporating these instance-aware mechanisms, InstaDrive achieves state-of-the-art video generation quality and enhances downstream autonomous driving tasks on the nuScenes dataset. Additionally, we utilize CARLA's autopilot to procedurally and stochastically simulate rare but safety-critical driving scenarios across diverse maps and regions, enabling rigorous safety evaluation for autonomous systems. Our project page is https://shanpoyang654.github.io/InstaDrive/page.html.

Figures (6)

• Figure 1: Limitations of Prior Works.(a) Temporal Consistency: In MagicDrive-V2 gao2024magicdrivedithighresolutionlongvideo, the car’s color changes inconsistently over time. (b) Spatial Localization: In MagicDrive-V2, the car deviates from the bounding box control signal. (c) Occlusion Hierarchy: In Panacea wen2024panacea, the distant bus incorrectly occludes the nearby pedestrian, violating natural occlusion rules. Our method excels in temporal consistency, spatial localization, and occlusion hierarchy, addressing these issues effectively.
• Figure 2: Overview. (a) Instance Flow Guider, which utilizes the instance flow to improve instance-level temporal consistency. (b) Spatial Geometric Aligner, which uses perspective projection and depth order to capture instance spatial locations and occlusion hierarchy. (c) Controlled Denoising Process, enabled by ST-DiT with ControlNet for unified condition control.
• Figure 3: Illustration of the extraction process for motion map in Instance Flow Guider. We calculate the motion vector $F_{\text{offset}}$ for each instance, and render the projected box using $F_{\text{offset}}$.
• Figure 4: Text control. By adding "Rainy," "Sunny," and "Night" to the original text prompt, while keeping other conditions unchanged, our model represents strong ability to edit videos effectively.
• Figure 5: Long-tail scenarios simulation. Using CARLA’s highly configurable simulation environment, we create synthetic control conditions which represent complex driving scenarios (e.g., sudden braking), and then utilize InstaDrive to generate corresponding videos.