From Forecasting to Planning: Policy World Model for Collaborative State-Action Prediction

TL;DR

This work introduces Policy World Model (PWM), a unified framework that couples world modeling with trajectory planning for autonomous driving by leveraging action-free future-state forecasting. PWM pretrains on large-scale video to learn world knowledge and then fine-tunes for planning using multimodal rationales, enabling planning directly from learned representations without action labels. A compact image tokenizer and a parallel token-generation scheme accelerate video forecasting, while Dynamic Focal Loss emphasizes temporally evolving regions to boost temporal dynamics. PWM achieves state-of-the-art or competitive results using only monocular front-camera input on nuScenes and NAVSIM, highlighting its potential for safer and more scalable autonomous systems.

Abstract

Despite remarkable progress in driving world models, their potential for autonomous systems remains largely untapped: the world models are mostly learned for world simulation and decoupled from trajectory planning. While recent efforts aim to unify world modeling and planning in a single framework, the synergistic facilitation mechanism of world modeling for planning still requires further exploration. In this work, we introduce a new driving paradigm named Policy World Model (PWM), which not only integrates world modeling and trajectory planning within a unified architecture, but is also able to benefit planning using the learned world knowledge through the proposed action-free future state forecasting scheme. Through collaborative state-action prediction, PWM can mimic the human-like anticipatory perception, yielding more reliable planning performance. To facilitate the efficiency of video forecasting, we further introduce a dynamically enhanced parallel token generation mechanism, equipped with a context-guided tokenizer and an adaptive dynamic focal loss. Despite utilizing only front camera input, our method matches or exceeds state-of-the-art approaches that rely on multi-view and multi-modal inputs. Code and model weights will be released at https://github.com/6550Zhao/Policy-World-Model.

Figures (11)

• Figure 1: Comparison of video world models for autonomous driving (a) Conventional video world models Adriver-iGaia-1VistaDrivingDojodrivinginoccworld typically serve as data engines for stimulation in pixel space. (b) Unified world models Drivinggptdoe-1vavim perform video generation and planning as separate tasks. (c) Our proposed policy world model performs planning based on the learned world knowledge.
• Figure 1: Detailed structure of input sequences. (a) illustrates the format of high-resolution and low-resolution video frames. (b) depicts the input configuration used for trajectory prediction.
• Figure 2: Policy World Model for Unified Forecasting and Planning. (a) PWM leverages its pre-trained world modeling to generate future frames, enabling seamless collaboration between perception, prediction, and planning. (b) Future video frames $\mathbf{I}_{1:N}$ are compressed into compact latent representations guided by initial frame $\mathbf{I}_{1}$ and $\hat{\mathbf{I}}_{1:N}$ represents the reconstructed frames.
• Figure 2: Visualzation. Comparison of future frame predictions with and without Dynamic Focal Loss (DFL). The first row shows ground truth frames, the second row shows predictions without DFL, and the third row shows predictions with DFL. Sampled frames at $t{=}2, 4, 6, 8, 10$ are shown.
• Figure 3: Pipeline for video world modeling. (a) World modeling is conducted on action-free, highly compressed video data using dynamically enhanced parallel prediction. (b) Comparison of token prediction formats and attention interactions.