Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

UniDrive-WM: Unified Understanding, Planning and Generation World Model For Autonomous Driving

Zhexiao Xiong, Xin Ye, Burhan Yaman, Sheng Cheng, Yiren Lu, Jingru Luo, Nathan Jacobs, Liu Ren

TL;DR

UniDrive-WM is proposed, a unified VLM-based world model that jointly performs driving-scene understanding, trajectory planning, and trajectory-conditioned future image generation within a single architecture and demonstrates the advantages of tightly integrating VLM-driven reasoning, planning, and generative world modeling for autonomous driving.

Abstract

World models have become central to autonomous driving, where accurate scene understanding and future prediction are crucial for safe control. Recent work has explored using vision-language models (VLMs) for planning, yet existing approaches typically treat perception, prediction, and planning as separate modules. We propose UniDrive-WM, a unified VLM-based world model that jointly performs driving-scene understanding, trajectory planning, and trajectory-conditioned future image generation within a single architecture. UniDrive-WM's trajectory planner predicts a future trajectory, which conditions a VLM-based image generator to produce plausible future frames. These predictions provide additional supervisory signals that enhance scene understanding and iteratively refine trajectory generation. We further compare discrete and continuous output representations for future image prediction, analyzing their influence on downstream driving performance. Experiments on the challenging Bench2Drive benchmark show that UniDrive-WM produces high-fidelity future images and improves planning performance by 5.9% in L2 trajectory error and 9.2% in collision rate over the previous best method. These results demonstrate the advantages of tightly integrating VLM-driven reasoning, planning, and generative world modeling for autonomous driving. The project page is available at https://unidrive-wm.github.io/UniDrive-WM .

UniDrive-WM: Unified Understanding, Planning and Generation World Model For Autonomous Driving

TL;DR

UniDrive-WM is proposed, a unified VLM-based world model that jointly performs driving-scene understanding, trajectory planning, and trajectory-conditioned future image generation within a single architecture and demonstrates the advantages of tightly integrating VLM-driven reasoning, planning, and generative world modeling for autonomous driving.

Abstract

World models have become central to autonomous driving, where accurate scene understanding and future prediction are crucial for safe control. Recent work has explored using vision-language models (VLMs) for planning, yet existing approaches typically treat perception, prediction, and planning as separate modules. We propose UniDrive-WM, a unified VLM-based world model that jointly performs driving-scene understanding, trajectory planning, and trajectory-conditioned future image generation within a single architecture. UniDrive-WM's trajectory planner predicts a future trajectory, which conditions a VLM-based image generator to produce plausible future frames. These predictions provide additional supervisory signals that enhance scene understanding and iteratively refine trajectory generation. We further compare discrete and continuous output representations for future image prediction, analyzing their influence on downstream driving performance. Experiments on the challenging Bench2Drive benchmark show that UniDrive-WM produces high-fidelity future images and improves planning performance by 5.9% in L2 trajectory error and 9.2% in collision rate over the previous best method. These results demonstrate the advantages of tightly integrating VLM-driven reasoning, planning, and generative world modeling for autonomous driving. The project page is available at https://unidrive-wm.github.io/UniDrive-WM .
Paper Structure (33 sections, 15 equations, 6 figures, 7 tables)

This paper contains 33 sections, 15 equations, 6 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. World Models for Autonomous Driving.
  4. Unified Image Understanding and Generation.
  5. Vision–Language–Action Models.
  6. Vision Language Model
  7. Vision Encoder
  8. Large Language Model.
  9. Trajectory Planner
  10. Future Image Generation
  11. Autoregressive Generation (Discrete Representation).
  12. AR+Diffusion Generation(Continuous Representation).
  13. Joint Image Generation and Planning
  14. Experiments
  15. Training Details
  16. ...and 18 more sections

Figures (6)

  • Figure 1: Top: trajectory-conditioned visual generation. Middle: VLM Instructed E2E Planning. Bottom: our unified future world modeling method. Compared with conditioned future image generation models and VLM-Instructed planning method, our framework establishes the connection between the reasoning, action and visual generation space via joint VLM-guided trajectory planner and future frame generation.
  • Figure 2: The pipeline of our UniDrive-WM Framework. The pipeline consists of: (1) a QT-Former based encoder to extract historical context and multi-view vision input; (2) The LLM for performing reasoning task and (3) The output layer generates the planning trajectory, future image prediction, which bridge the gaps between the planning space, image space and reasoning space. For the output layer, we provide detailed analysis in Fig. \ref{['fig:img_gen']}.
  • Figure 3: The two design choices for image generation in unified multimodal model. For the future image prediction, we use both Autoregressive and Autoregressive+Diffusion architecture. (a) Left: Autoregressive architecture; (b) Right: AR+Diffusion architecture.
  • Figure 4: Visualization about autoregressive future image prediction results. Left: Current frame; Middle: Future frame; Right: Predicted future image.
  • Figure 5: Visualization of AR+Diffusion result. Left: Current frame; Middle: Future frame; Right: Predicted future image.
  • ...and 1 more figures