DriveGen3D: Boosting Feed-Forward Driving Scene Generation with Efficient Video Diffusion

TL;DR

DriveGen3D tackles the challenge of efficiently generating long-horizon driving videos and the corresponding dynamic 3D scenes under multimodal control. It introduces FastDrive-DiT, which accelerates high-resolution diffusion-based video synthesis via diffusion-step caching and quantized attention, and FastRecon3D, a feed-forward temporal Gaussian splatting module for fast 3D scene reconstruction. The combined system achieves real-time-like performance (up to $424\times800$ at 12 FPS) and reconstruction SSIM of $0.811$ with PSNR $22.84$, while significantly reducing end-to-end generation time to about 6 minutes. This work advances autonomous-driving simulation and dynamic world modeling by offering a scalable, efficient alternative to voxel-based or optimization-heavy pipelines.

Abstract

We present DriveGen3D, a novel framework for generating high-quality and highly controllable dynamic 3D driving scenes that addresses critical limitations in existing methodologies. Current approaches to driving scene synthesis either suffer from prohibitive computational demands for extended temporal generation, focus exclusively on prolonged video synthesis without 3D representation, or restrict themselves to static single-scene reconstruction. Our work bridges this methodological gap by integrating accelerated long-term video generation with large-scale dynamic scene reconstruction through multimodal conditional control. DriveGen3D introduces a unified pipeline consisting of two specialized components: FastDrive-DiT, an efficient video diffusion transformer for high-resolution, temporally coherent video synthesis under text and Bird's-Eye-View (BEV) layout guidance; and FastRecon3D, a feed-forward reconstruction module that rapidly builds 3D Gaussian representations across time, ensuring spatial-temporal consistency. Together, these components enable real-time generation of extended driving videos (up to $424\times800$ at 12 FPS) and corresponding dynamic 3D scenes, achieving SSIM of 0.811 and PSNR of 22.84 on novel view synthesis, all while maintaining parameter efficiency.

Figures (7)

• Figure 1: DriveGen3D visualization. Comparison of GT videos and generated video for reconstruction.
• Figure 2: Performance vs. Efficiency. DriveGen3D achieves the highest SSIM (0.811) while significantly reducing generation time to 6 minutes—an 80% improvement over optimization-based and diffusion-based baselines—demonstrating superior video quality and real-time capability.
• Figure 3: Overview of DriveGen3D. (a) Given textual and BEV layout conditions, our model first employs an accelerated Video Diffusion Transformer to synthesize a long driving video. (b) Next, a per-frame 3D Gaussian Splatting representation is utilized to construct entire scene from the generated video frames.
• Figure 4: Typical examples of tensors’ data distribution in different attention blocks of MagicDriveDiT.
• Figure 5: Visualization of input differences and output differences in consecutive timesteps. We plot all, condition and uncondition separetely.