OmniNWM: Omniscient Driving Navigation World Models

TL;DR

OmniNWM addresses the need for unified, long-horizon driving world models by jointly forecasting panoramic RGB, semantics, depth, and 3D occupancy while enabling precise, pixel-level action control through normalized Plücker ray-maps. It introduces occupancy-grounded dense rewards to support closed-loop evaluation and planning, and a flexible forcing strategy to maintain generation quality over extended horizons, achieving state-of-the-art results in generation fidelity and control accuracy. The framework is complemented by OmniNWM-VLA and Tri-MIDI for planning, plus a Panoramic Diffusion Transformer that integrates multi-modal signals. Together, these elements yield strong zero-shot generalization and robust, occupancy-informed evaluation, highlighting the practical potential for holistic driving system development and testing.

Abstract

Autonomous driving world models are expected to work effectively across three core dimensions: state, action, and reward. Existing models, however, are typically restricted to limited state modalities, short video sequences, imprecise action control, and a lack of reward awareness. In this paper, we introduce OmniNWM, an omniscient panoramic navigation world model that addresses all three dimensions within a unified framework. For state, OmniNWM jointly generates panoramic videos of RGB, semantics, metric depth, and 3D occupancy. A flexible forcing strategy enables high-quality long-horizon auto-regressive generation. For action, we introduce a normalized panoramic Plucker ray-map representation that encodes input trajectories into pixel-level signals, enabling highly precise and generalizable control over panoramic video generation. Regarding reward, we move beyond learning reward functions with external image-based models: instead, we leverage the generated 3D occupancy to directly define rule-based dense rewards for driving compliance and safety. Extensive experiments demonstrate that OmniNWM achieves state-of-the-art performance in video generation, control accuracy, and long-horizon stability, while providing a reliable closed-loop evaluation framework through occupancy-grounded rewards. Project page is available at https://arlo0o.github.io/OmniNWM/.

Figures (19)

• Figure 1: Versatile capabilities of OmniNWM. (a) State: Given a conditional image and ego trajectory, OmniNWM jointly generates comprehensive multi-modal outputs, including panoramic RGB, semantic, metric depth, and 3D occupancy videos. (b) Action: Conditioned on different input trajectories, OmniNWM facilitates precise panoramic camera control by converting them into normalized Plücker ray-maps as pixel-level guidance. (c) Reward: OmniNWM enables unbounded long-term navigation through a closed-loop pipeline: the planning trajectory guides the multi-modal generation, while dense rewards are derived from the generated 3D semantic occupancy. Note that all input trajectories are encoded into normalized Plücker ray-maps before guiding the generation process, even when not explicitly visualized.
• Figure 2: Architecture overview of OmniNWM. Our model takes as input a historical trajectory, a reference image, and flexible multi-level noise applied view-wise and frame-wise. It simultaneously forecasts panoramic RGB, semantic, and metric depth videos (Section \ref{['sec_pixel']}), along with 3D occupancy for intrinsic dense rewards (Section \ref{['sec_occupancy']}) and future planning trajectories (Section \ref{['sec_trajectory']}). Note that input trajectories guide the generation of the current video clip, while future trajectories are forecasted for the next clip, forming a closed-loop system. The input trajectories with camera poses are encoded into normalized panoramic Plücker ray-maps, providing a unified pixel-level description and enabling precise action control (Section \ref{['sec_raymap']}). A flexible forcing strategy injects independent multi-level noise at the view and frame levels to support robust and flexible auto-regressive generation (Section \ref{['sec_flexible']}).
• Figure 3: Occupancy generation. The generated panoramic RGB images, semantics, and depth are used to produce a 3D voxel volume. This design enables lightweight occupancy prediction compared with prior volumetric-based methods li2025uniscenezheng2023occworld, while naturally supporting the integration of occupancy-grounded rewards.
• Figure 4: Average rewards of different trajectories, computed using the proposed 3D occupancy-grounded reward function. The rewards effectively evaluate the feasibility of planning trajectories in the presence of obstacles (e.g., an oncoming truck).
• Figure 5: Panoramic normalized ray-map encoding. (a) Plücker ray-maps derived from panoramic camera poses are normalized in both scale and pose. (b) The ray-map normalization process constructs different trajectories within a unified 3D Plücker space. (c) Compared to the original NuScenes dataset, our strategy significantly enhances the diversity of trajectory distributions.