Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Latent-WAM: Latent World Action Modeling for End-to-End Autonomous Driving

Linbo Wang, Yupeng Zheng, Qiang Chen, Shiwei Li, Yichen Zhang, Zebin Xing, Qichao Zhang, Xiang Li, Deheng Qian, Pengxuan Yang, Yihang Dong, Ce Hao, Xiaoqing Ye, Junyu han, Yifeng Pan, Dongbin Zhao

Abstract

We introduce Latent-WAM, an efficient end-to-end autonomous driving framework that achieves strong trajectory planning through spatially-aware and dynamics-informed latent world representations. Existing world-model-based planners suffer from inadequately compressed representations, limited spatial understanding, and underutilized temporal dynamics, resulting in sub-optimal planning under constrained data and compute budgets. Latent-WAM addresses these limitations with two core modules: a Spatial-Aware Compressive World Encoder (SCWE) that distills geometric knowledge from a foundation model and compresses multi-view images into compact scene tokens via learnable queries, and a Dynamic Latent World Model (DLWM) that employs a causal Transformer to autoregressively predict future world status conditioned on historical visual and motion representations. Extensive experiments on NAVSIM v2 and HUGSIM demonstrate new state-of-the-art results: 89.3 EPDMS on NAVSIM v2 and 28.9 HD-Score on HUGSIM, surpassing the best prior perception-free method by 3.2 EPDMS with significantly less training data and a compact 104M-parameter model.

Latent-WAM: Latent World Action Modeling for End-to-End Autonomous Driving

Abstract

We introduce Latent-WAM, an efficient end-to-end autonomous driving framework that achieves strong trajectory planning through spatially-aware and dynamics-informed latent world representations. Existing world-model-based planners suffer from inadequately compressed representations, limited spatial understanding, and underutilized temporal dynamics, resulting in sub-optimal planning under constrained data and compute budgets. Latent-WAM addresses these limitations with two core modules: a Spatial-Aware Compressive World Encoder (SCWE) that distills geometric knowledge from a foundation model and compresses multi-view images into compact scene tokens via learnable queries, and a Dynamic Latent World Model (DLWM) that employs a causal Transformer to autoregressively predict future world status conditioned on historical visual and motion representations. Extensive experiments on NAVSIM v2 and HUGSIM demonstrate new state-of-the-art results: 89.3 EPDMS on NAVSIM v2 and 28.9 HD-Score on HUGSIM, surpassing the best prior perception-free method by 3.2 EPDMS with significantly less training data and a compact 104M-parameter model.

Paper Structure

This paper contains 33 sections, 18 equations, 15 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Scene Representation in End-to-End Autonomous Driving
  4. World Models for Autonomous Driving
  5. Method
  6. Overall
  7. Spatial-Aware Compressive World Encoder
  8. Scene Compression.
  9. Geometric Alignment.
  10. Dynamic Latent World Model
  11. World latent status aggregation.
  12. Causal world model prediction.
  13. Teacher Forcing Attention Mask.
  14. 3D-RoPE.
  15. Ego status supervision.
  16. ...and 18 more sections

Figures (15)

  • Figure 1: Performance vs. training data scale on NAVSIM v2. Bubble size indicates model parameters. Our Latent-WAM achieves the highest EPDMS with significantly fewer training data and smaller model size, demonstrating superior data efficiency over existing world-model-based methods. World4Drive is marked separately as it employs an additional ViT-L depth estimator.
  • Figure 2: Overview of the Latent-WAM architecture. See \ref{['sect:overall']} for details.
  • Figure 3: Teacher Forcing Attention Mask.
  • Figure 4: Visualization of planning trajectories, where the green line is the human trajectory, the yellow line is the predicted trajectory of corresponding method.
  • Figure 5: Visualization of attention maps between scene tokens and image patches. From top to bottom, the three groups correspond to going straight, turning right, and turning left respectively.
  • ...and 10 more figures