Vision-Centric 4D Occupancy Forecasting and Planning via Implicit Residual World Models

TL;DR

IR-WM introduces an Implicit Residual World Model that reframes vision-centric autonomous driving from full future scene reconstruction to modeling the current BEV state plus residual, action-conditioned changes. A streaming-memory transformer predicts residual BEV updates, which are aligned and combined with prior BEV features to forecast 4D occupancy and guide planning. The method uses an occupancy-based supervisory signal and explores different forecasting–planning coupling schemes, finding that forecasting the implicit future state substantially improves planning accuracy while filtering by occupancy offers limited gains. Experiments on nuScenes/OpenOccupancy show state-of-the-art performance in both 4D occupancy forecasting and end-to-end planning, with robust generalization to static and dynamic scene elements and improved safety metrics. Overall, IR-WM presents a data-efficient, temporally coherent framework that enhances planning by leveraging a compact, residual, vision-based world model.

Abstract

End-to-end autonomous driving systems increasingly rely on vision-centric world models to understand and predict their environment. However, a common ineffectiveness in these models is the full reconstruction of future scenes, which expends significant capacity on redundantly modeling static backgrounds. To address this, we propose IR-WM, an Implicit Residual World Model that focuses on modeling the current state and evolution of the world. IR-WM first establishes a robust bird's-eye-view representation of the current state from the visual observation. It then leverages the BEV features from the previous timestep as a strong temporal prior and predicts only the "residual", i.e., the changes conditioned on the ego-vehicle's actions and scene context. To alleviate error accumulation over time, we further apply an alignment module to calibrate semantic and dynamic misalignments. Moreover, we investigate different forecasting-planning coupling schemes and demonstrate that the implicit future state generated by world models substantially improves planning accuracy. On the nuScenes benchmark, IR-WM achieves top performance in both 4D occupancy forecasting and trajectory planning.

Figures (5)

• Figure 1: We model the current state and predict residual future changes, yielding a more effective allocation of model capacity.
• Figure 2: Overview of IR-WM. IR-WM comprises three parts: (a) Scene encoding from visual observations via a scene encoder; (b) future forecasting, performed in an implicit residual prediction manner with a streaming memory using an autoregressive future predictor; (c) Task decoding via occupancy head $D_o$ and planning head $D_p$. "FA" denotes the feature alignment for alleviating error accumulation.
• Figure 3: The detailed architecture of the planning head. "Adapter" is a stack of convolutions.
• Figure 4: Three variants illustrating how forecasting affects planning; the blue line marks components that can be removed during inference.
• Figure 5: Qualitative results for 4D occupancy forecasting and planning. IR-WM produces reliable predictions for both static and dynamic objects across different times of day.