Driving in the Occupancy World: Vision-Centric 4D Occupancy Forecasting and Planning via World Models for Autonomous Driving

TL;DR

Drive-OccWorld introduces a vision-centric 4D occupancy forecasting framework that links future state prediction with end-to-end planning for autonomous driving. It integrates a history BEV encoder, a memory queue with semantic- and motion-conditional normalization, and a transformer-based world decoder that accepts diverse action conditions to produce controllable futures. An occupancy-based planner then selects safe trajectories using a composite cost function, enabling continuous forecasting and planning. Empirical results on nuScenes, nuScenes-Occupancy, and Lyft-Level5 demonstrate state-of-the-art forecasting performance and improved planning robustness, validating the approach's potential for safe, explainable driving in dynamic scenes.

Abstract

World models envision potential future states based on various ego actions. They embed extensive knowledge about the driving environment, facilitating safe and scalable autonomous driving. Most existing methods primarily focus on either data generation or the pretraining paradigms of world models. Unlike the aforementioned prior works, we propose Drive-OccWorld, which adapts a vision-centric 4D forecasting world model to end-to-end planning for autonomous driving. Specifically, we first introduce a semantic and motion-conditional normalization in the memory module, which accumulates semantic and dynamic information from historical BEV embeddings. These BEV features are then conveyed to the world decoder for future occupancy and flow forecasting, considering both geometry and spatiotemporal modeling. Additionally, we propose injecting flexible action conditions, such as velocity, steering angle, trajectory, and commands, into the world model to enable controllable generation and facilitate a broader range of downstream applications. Furthermore, we explore integrating the generative capabilities of the 4D world model with end-to-end planning, enabling continuous forecasting of future states and the selection of optimal trajectories using an occupancy-based cost function. Comprehensive experiments conducted on the nuScenes, nuScenes-Occupancy, and Lyft-Level5 datasets illustrate that our method can generate plausible and controllable 4D occupancy, paving the way for advancements in driving world generation and end-to-end planning. Project page: https://drive-occworld.github.io/

Figures (10)

• Figure 1: 4D Occupancy Forecasting and Planning via World Model. Drive-OccWorld takes observations and trajectories as input, incorporating flexible action conditions for action-controllable generation. By leveraging world knowledge and the generative capacity of the world model, we further integrate it with a planner for continuous forecasting and planning.
• Figure 2: Overview of Drive-OccWorld. (a) The history encoder extracts multi-view image features and transforms them into BEV embeddings. (b) The memory queue employs semantic- and motion-conditional normalization to aggregate historical information. (c) The world decoder incorporates action conditions to generate various future occupancies and flows. Integrating the world decoder with an occupancy-based planner enables continuous forecasting and planning.
• Figure 3: Overview of semantic-conditional normalization.
• Figure 4: Qualitative results of 4D occupancy and flow forecasting. The results are presented at various future timestamps.
• Figure 5: Qualitative results of controllable generation, using the high-level command or low-level trajectory conditions.