Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DiVE: Efficient Multi-View Driving Scenes Generation Based on Video Diffusion Transformer

Junpeng Jiang, Gangyi Hong, Miao Zhang, Hengtong Hu, Kun Zhan, Rui Shao, Liqiang Nie

TL;DR

DiVE introduces a DiT-based diffusion framework for controllable multi-view driving scene video generation, integrating text, road sketches, 3D object constraints, and camera parameters via unified cross-attention and view-inflated mechanisms. Its SketchFormer road encoding and multi-scale training ensure BEV-aligned geometric consistency and temporal cross-view coherence. To address classifier-free guidance complexity and high-resolution latency, the authors propose Multi-Control Auxiliary Branch Distillation (MAD) and Resolution Progressively Sampling (RPS), achieving a 2.62x inference speedup with minimal quality loss. On nuScenes, DiVE achieves state-of-the-art generative quality and cross-view/temporal coherence, and synthetic data generated by DiVE substantially boosts downstream perception models, underscoring the practical value of controllable, high-fidelity synthetic driving scenes.

Abstract

Collecting multi-view driving scenario videos to enhance the performance of 3D visual perception tasks presents significant challenges and incurs substantial costs, making generative models for realistic data an appealing alternative. Yet, the videos generated by recent works suffer from poor quality and spatiotemporal consistency, undermining their utility in advancing perception tasks under driving scenarios. To address this gap, we propose DiVE, a diffusion transformer-based generative framework meticulously engineered to produce high-fidelity, temporally coherent, and cross-view consistent multi-view videos, aligning seamlessly with bird's-eye view layouts and textual descriptions. DiVE leverages a unified cross-attention and a SketchFormer to exert precise control over multimodal data, while incorporating a view-inflated attention mechanism that adds no extra parameters, thereby guaranteeing consistency across views. Despite these advancements, synthesizing high-resolution videos under multimodal constraints introduces dual challenges: investigating the optimal classifier-free guidance coniguration under intricate multi-condition inputs and mitigating excessive computational latency in high-resolution rendering--both of which remain underexplored in prior researches. To resolve these limitations, we introduce two innovations: Multi-Control Auxiliary Branch Distillation, which streamlines multi-condition CFG selection while circumventing high computational overhead, and Resolution Progressive Sampling, a training-free acceleration strategy that staggers resolution scaling to reduce high latency due to high resolution. These innovations collectively achieve a 2.62x speedup with minimal quality degradation. Evaluated on the nuScenes dataset, DiVE achieves SOTA performance in multi-view video generation, yielding photorealistic outputs with exceptional temporal and cross-view coherence.

DiVE: Efficient Multi-View Driving Scenes Generation Based on Video Diffusion Transformer

TL;DR

DiVE introduces a DiT-based diffusion framework for controllable multi-view driving scene video generation, integrating text, road sketches, 3D object constraints, and camera parameters via unified cross-attention and view-inflated mechanisms. Its SketchFormer road encoding and multi-scale training ensure BEV-aligned geometric consistency and temporal cross-view coherence. To address classifier-free guidance complexity and high-resolution latency, the authors propose Multi-Control Auxiliary Branch Distillation (MAD) and Resolution Progressively Sampling (RPS), achieving a 2.62x inference speedup with minimal quality loss. On nuScenes, DiVE achieves state-of-the-art generative quality and cross-view/temporal coherence, and synthetic data generated by DiVE substantially boosts downstream perception models, underscoring the practical value of controllable, high-fidelity synthetic driving scenes.

Abstract

Collecting multi-view driving scenario videos to enhance the performance of 3D visual perception tasks presents significant challenges and incurs substantial costs, making generative models for realistic data an appealing alternative. Yet, the videos generated by recent works suffer from poor quality and spatiotemporal consistency, undermining their utility in advancing perception tasks under driving scenarios. To address this gap, we propose DiVE, a diffusion transformer-based generative framework meticulously engineered to produce high-fidelity, temporally coherent, and cross-view consistent multi-view videos, aligning seamlessly with bird's-eye view layouts and textual descriptions. DiVE leverages a unified cross-attention and a SketchFormer to exert precise control over multimodal data, while incorporating a view-inflated attention mechanism that adds no extra parameters, thereby guaranteeing consistency across views. Despite these advancements, synthesizing high-resolution videos under multimodal constraints introduces dual challenges: investigating the optimal classifier-free guidance coniguration under intricate multi-condition inputs and mitigating excessive computational latency in high-resolution rendering--both of which remain underexplored in prior researches. To resolve these limitations, we introduce two innovations: Multi-Control Auxiliary Branch Distillation, which streamlines multi-condition CFG selection while circumventing high computational overhead, and Resolution Progressive Sampling, a training-free acceleration strategy that staggers resolution scaling to reduce high latency due to high resolution. These innovations collectively achieve a 2.62x speedup with minimal quality degradation. Evaluated on the nuScenes dataset, DiVE achieves SOTA performance in multi-view video generation, yielding photorealistic outputs with exceptional temporal and cross-view coherence.
Paper Structure (28 sections, 3 equations, 13 figures, 5 tables, 3 algorithms)

This paper contains 28 sections, 3 equations, 13 figures, 5 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. DiVE
  5. Multi-Control Auxiliary Branch Distillation
  6. Resolution Progressively Sampling
  7. Experiments
  8. Experimental Setups
  9. Implementation Details.
  10. Main Results
  11. Ablation Study
  12. More Results
  13. Conclusion
  14. More Related Work
  15. Diffusion Transformer for Generation
  16. ...and 13 more sections

Figures (13)

  • Figure 1: Overview of DiVE for multi-view video generation. Our model encodes four inputs for controllable generation: scene description words for global context, camera information for motion control, bounding boxes that locate 3D objects placement, and road sketches for road conditions. Each block in DiVE consists of spatial attention, temporal attention, cross attention, and an MLP. Notably, the view-inflated attention, which enhances view consistency, is integrated into the spatial attention mechanism within the backbone network.
  • Figure 2: The overall process of Multi-Control Auxiliary Branch Distillation. CA denotes cross-attention.
  • Figure 3: The overall process of Resolution Progressively Sampling. Larger quadrilaterals represent higher resolutions, and deeper colors indicate more noisy regions.
  • Figure 4: Visualization of quantitative comparison.
  • Figure 5: Qualitative comparison of DiVE with MagicDrive and Panacea. We use dashed boxes to highlight some of the noticeable issues in MagicDrive and Panacea, and arrows to indicate the changes in vehicle positions over time. In contrast, DiVE demonstrates superior realism, temporal and cross-view consistency, and controllability, both before and after applying RPS or MAD.
  • ...and 8 more figures