Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

UniDriveDreamer: A Single-Stage Multimodal World Model for Autonomous Driving

Guosheng Zhao, Yaozeng Wang, Xiaofeng Wang, Zheng Zhu, Tingdong Yu, Guan Huang, Yongchen Zai, Ji Jiao, Changliang Xue, Xiaole Wang, Zhen Yang, Futang Zhu, Xingang Wang

TL;DR

This paper tackles the challenge of multimodal data synthesis for autonomous driving by moving beyond single-modality or cascaded multimodal generation. It introduces UniDriveDreamer, a single-stage world model that jointly generates multi-view camera videos and LiDAR sweeps using two modality-specific VAEs, a layout and text encoder, and a diffusion transformer, with Unified Latent Anchoring to align cross-modal latents. The approach achieves state-of-the-art results on both video and LiDAR generation metrics and yields measurable downstream perception improvements, demonstrating the practical value of coherent multimodal synthesis. By enabling temporally consistent and geometrically accurate multimodal outputs without intermediate representations, this work offers a more data-efficient path for sensor simulation and autonomous driving development.

Abstract

World models have demonstrated significant promise for data synthesis in autonomous driving. However, existing methods predominantly concentrate on single-modality generation, typically focusing on either multi-camera video or LiDAR sequence synthesis. In this paper, we propose UniDriveDreamer, a single-stage unified multimodal world model for autonomous driving, which directly generates multimodal future observations without relying on intermediate representations or cascaded modules. Our framework introduces a LiDAR-specific variational autoencoder (VAE) designed to encode input LiDAR sequences, alongside a video VAE for multi-camera images. To ensure cross-modal compatibility and training stability, we propose Unified Latent Anchoring (ULA), which explicitly aligns the latent distributions of the two modalities. The aligned features are fused and processed by a diffusion transformer that jointly models their geometric correspondence and temporal evolution. Additionally, structured scene layout information is projected per modality as a conditioning signal to guide the synthesis. Extensive experiments demonstrate that UniDriveDreamer outperforms previous state-of-the-art methods in both video and LiDAR generation, while also yielding measurable improvements in downstream

UniDriveDreamer: A Single-Stage Multimodal World Model for Autonomous Driving

TL;DR

This paper tackles the challenge of multimodal data synthesis for autonomous driving by moving beyond single-modality or cascaded multimodal generation. It introduces UniDriveDreamer, a single-stage world model that jointly generates multi-view camera videos and LiDAR sweeps using two modality-specific VAEs, a layout and text encoder, and a diffusion transformer, with Unified Latent Anchoring to align cross-modal latents. The approach achieves state-of-the-art results on both video and LiDAR generation metrics and yields measurable downstream perception improvements, demonstrating the practical value of coherent multimodal synthesis. By enabling temporally consistent and geometrically accurate multimodal outputs without intermediate representations, this work offers a more data-efficient path for sensor simulation and autonomous driving development.

Abstract

World models have demonstrated significant promise for data synthesis in autonomous driving. However, existing methods predominantly concentrate on single-modality generation, typically focusing on either multi-camera video or LiDAR sequence synthesis. In this paper, we propose UniDriveDreamer, a single-stage unified multimodal world model for autonomous driving, which directly generates multimodal future observations without relying on intermediate representations or cascaded modules. Our framework introduces a LiDAR-specific variational autoencoder (VAE) designed to encode input LiDAR sequences, alongside a video VAE for multi-camera images. To ensure cross-modal compatibility and training stability, we propose Unified Latent Anchoring (ULA), which explicitly aligns the latent distributions of the two modalities. The aligned features are fused and processed by a diffusion transformer that jointly models their geometric correspondence and temporal evolution. Additionally, structured scene layout information is projected per modality as a conditioning signal to guide the synthesis. Extensive experiments demonstrate that UniDriveDreamer outperforms previous state-of-the-art methods in both video and LiDAR generation, while also yielding measurable improvements in downstream
Paper Structure (12 sections, 5 equations, 7 figures, 4 tables)

This paper contains 12 sections, 5 equations, 7 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Unimodal Driving Scene Generation
  4. Multimodal Driving Scene Generation
  5. UniDriveDreamer
  6. LiDAR VAE
  7. Single-Stage Multimodal World Model
  8. Experiment
  9. Experiment Setup
  10. Main Results
  11. Ablation Study
  12. Conclusion

Figures (7)

  • Figure 1: The overall framework of UniDriveDreamer. Our UniDriveDreamer consists of four core components: (1) two modality-specific VAEs that encode multi-view camera images and LiDAR range maps into a shared latent space; (2) a layout encoder that projects structured scene layout information into corresponding latent representations; (3) a text encoder that encode multi-view text prompts into corresponding prompt embeddings; and (4) a diffusion transformer that jointly models spatiotemporal coherence within each modality and cross-modal consistency across modalities.
  • Figure 2: Transformer block of UniDriveDreamer.
  • Figure 3: Qualitative comparison of generated outputs with Genesis genesis. The top row shows the ground truth. The middle row presents results from Genesis. The bottom row displays outputs from our UniDriveDreamer.
  • Figure 4: Qualitative visualization of multimodal outputs generated by UniDriveDreamer. In each example set, the upper row presents the ground truth, while the lower row shows the corresponding synthesized results generated by UniDriveDreamer.
  • Figure 5: Qualitative visualization of multimodal outputs generated by UniDriveDreamer. In each example set, the upper row presents the ground truth, while the lower row shows the corresponding synthesized results generated by UniDriveDreamer.
  • ...and 2 more figures