Epona: Autoregressive Diffusion World Model for Autonomous Driving

TL;DR

Problem: existing driving world models struggle to produce long-horizon, high-fidelity predictions while supporting planning. Approach: Epona uses autoregressive diffusion with decoupled spatiotemporal factorization and asynchronous multi-modal generation (TrajDiT for trajectories and VisDiT for frames), plus a chain-of-forward training strategy. Contributions: achieves minutes-long video generation with state-of-the-art FVD on NuScenes, enables real-time motion planning on NAVSIM without perception inputs, and demonstrates self-supervised acquisition of driving knowledge from future prediction. Significance: provides a practical, end-to-end driving world model that couples planning and visual modeling for robust autonomous driving.

Abstract

Diffusion models have demonstrated exceptional visual quality in video generation, making them promising for autonomous driving world modeling. However, existing video diffusion-based world models struggle with flexible-length, long-horizon predictions and integrating trajectory planning. This is because conventional video diffusion models rely on global joint distribution modeling of fixed-length frame sequences rather than sequentially constructing localized distributions at each timestep. In this work, we propose Epona, an autoregressive diffusion world model that enables localized spatiotemporal distribution modeling through two key innovations: 1) Decoupled spatiotemporal factorization that separates temporal dynamics modeling from fine-grained future world generation, and 2) Modular trajectory and video prediction that seamlessly integrate motion planning with visual modeling in an end-to-end framework. Our architecture enables high-resolution, long-duration generation while introducing a novel chain-of-forward training strategy to address error accumulation in autoregressive loops. Experimental results demonstrate state-of-the-art performance with 7.4\% FVD improvement and minutes longer prediction duration compared to prior works. The learned world model further serves as a real-time motion planner, outperforming strong end-to-end planners on NAVSIM benchmarks. Code will be publicly available at \href{https://github.com/Kevin-thu/Epona/}{https://github.com/Kevin-thu/Epona/}.

Figures (10)

• Figure 1: Versatile capabilities of Epona. Given historical driving context, our Epona can generate consistent minutes-long future driving scenes at high resolution (A). It can be controlled by diverse trajectories (B), and understand real-world traffic knowledge (C). In addition, our world model can predict future trajectories and serve as an end-to-end real-time motion planner (D).
• Figure 2: Overview of Epona. Our world model utilizes a multimodal spatiotemporal transformer to process the historical context of the first $T$ frames and employs a next-frame prediction DiT to generate the frame at $T+1$ and a trajectory planning DiT to forecast the future $N$-frame pose trajectory. By adopting a chain-of-forward strategy, our approach enables high-quality and long-horizon video generation with an autoregressive manner.
• Figure 3: Comparison of Different World Modeling Formulation. Up: Conventional autoregressive pipeline quantizes continous images into discrete tokens and perform next-token prediction iteratively. Middle: The video-diffusion-based methods generate future $n$ frames simultaneously. Down: Our method autoregressively predicts fine-grained future frames in continuous space.
• Figure 4: Concept illustration of our training process. Here $x$ can be either image latents or trajectories.
• Figure 5: Qualitative Comparison between Vista gao2024vista and Epona. Zoom in for better views.