Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

UniDWM: Towards a Unified Driving World Model via Multifaceted Representation Learning

Shuai Liu, Siheng Ren, Xiaoyao Zhu, Quanmin Liang, Zefeng Li, Qiang Li, Xin Hu, Kai Huang

TL;DR

UniDWM tackles the problem of unified driving intelligence by learning a structure- and dynamics-aware latent world model that jointly supports perception, prediction, and planning from partial visual observations. It introduces a joint reconstruction pathway to recover geometry, appearance, and ego-motion, and a diffusion-based collaborative generation module to forecast future latent evolution, grounded in an InfoVAE-style objective. The approach is validated on the NAVSIM dataset, where it demonstrates improved trajectory planning, superior 4D reconstruction accuracy, and coherent 4D generation, all while reducing reliance on dense perception annotations. This work provides a principled, annotation-efficient framework for multifaceted world representations and highlights the potential of unified latent spaces as a foundation for driving intelligence.

Abstract

Achieving reliable and efficient planning in complex driving environments requires a model that can reason over the scene's geometry, appearance, and dynamics. We present UniDWM, a unified driving world model that advances autonomous driving through multifaceted representation learning. UniDWM constructs a structure- and dynamic-aware latent world representation that serves as a physically grounded state space, enabling consistent reasoning across perception, prediction, and planning. Specifically, a joint reconstruction pathway learns to recover the scene's structure, including geometry and visual texture, while a collaborative generation framework leverages a conditional diffusion transformer to forecast future world evolution within the latent space. Furthermore, we show that our UniDWM can be deemed as a variation of VAE, which provides theoretical guidance for the multifaceted representation learning. Extensive experiments demonstrate the effectiveness of UniDWM in trajectory planning, 4D reconstruction and generation, highlighting the potential of multifaceted world representations as a foundation for unified driving intelligence. The code will be publicly available at https://github.com/Say2L/UniDWM.

UniDWM: Towards a Unified Driving World Model via Multifaceted Representation Learning

TL;DR

UniDWM tackles the problem of unified driving intelligence by learning a structure- and dynamics-aware latent world model that jointly supports perception, prediction, and planning from partial visual observations. It introduces a joint reconstruction pathway to recover geometry, appearance, and ego-motion, and a diffusion-based collaborative generation module to forecast future latent evolution, grounded in an InfoVAE-style objective. The approach is validated on the NAVSIM dataset, where it demonstrates improved trajectory planning, superior 4D reconstruction accuracy, and coherent 4D generation, all while reducing reliance on dense perception annotations. This work provides a principled, annotation-efficient framework for multifaceted world representations and highlights the potential of unified latent spaces as a foundation for driving intelligence.

Abstract

Achieving reliable and efficient planning in complex driving environments requires a model that can reason over the scene's geometry, appearance, and dynamics. We present UniDWM, a unified driving world model that advances autonomous driving through multifaceted representation learning. UniDWM constructs a structure- and dynamic-aware latent world representation that serves as a physically grounded state space, enabling consistent reasoning across perception, prediction, and planning. Specifically, a joint reconstruction pathway learns to recover the scene's structure, including geometry and visual texture, while a collaborative generation framework leverages a conditional diffusion transformer to forecast future world evolution within the latent space. Furthermore, we show that our UniDWM can be deemed as a variation of VAE, which provides theoretical guidance for the multifaceted representation learning. Extensive experiments demonstrate the effectiveness of UniDWM in trajectory planning, 4D reconstruction and generation, highlighting the potential of multifaceted world representations as a foundation for unified driving intelligence. The code will be publicly available at https://github.com/Say2L/UniDWM.
Paper Structure (24 sections, 31 equations, 4 figures, 5 tables)

This paper contains 24 sections, 31 equations, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Preliminary
  5. Unified Driving World Model
  6. Framework of UniDWM
  7. Joint Reconstruction
  8. Collaborative Generation
  9. Experiment
  10. Implementation Details
  11. Experimental Results
  12. Ablation Study
  13. Conclusion
  14. Proof of VAE-style ELBO Objective of UniDWM
  15. Problem Setup
  16. ...and 9 more sections

Figures (4)

  • Figure 1: The overall framework of our UniDWM. UniDWM is composed of the Reconstruction and Generation stages. Given a sequence of visual observations as input, UniDWM projects them to a unified feature space using static and dynamic encoders. Then, reconstruction and generation decoders can be used to reconstruct the observed scene and generate dynamic evolution, respectively.
  • Figure 2: Architecture of the DiT block, consisting of an alternating sequence of spatial DiT (S-DiT) and temporal DiT (T-DiT) modules.
  • Figure 3: Visualization of 4D recontruction results. The left images are the sequential visual inputs, the right visualized points are the reconstructed geometry.
  • Figure 4: Visualization of 4D generation results. Given sequential prior visual frames, UniDWM generates future visual frames with corresponding geometry. We merge the geometries and visualize them as colored points.