An Efficient Occupancy World Model via Decoupled Dynamic Flow and Image-assisted Training

TL;DR

This work introduces DFIT-OccWorld, an end-to-end, non-autoregressive 3D occupancy world model that forecasts future scenes by decoupling dynamic voxel flow from static voxel occupancy and warping, with image-assisted training to guide learning. By replacing the prior two-stage training of OccWorld with a single-stage pipeline and a cross-modal SALT-based encoder, the method achieves state-of-the-art 4D occupancy forecasting and improves motion planning and point-cloud forecasting on nuScenes and OpenScene. Key innovations include decoupled dynamic flow for efficient prediction, voxel warping for occupancy synthesis, and a rendering-based photometric consistency loss to leverage image data during training. The approach yields large accuracy gains and significant efficiency improvements, making it practical for real-time autonomous driving applications.

Abstract

The field of autonomous driving is experiencing a surge of interest in world models, which aim to predict potential future scenarios based on historical observations. In this paper, we introduce DFIT-OccWorld, an efficient 3D occupancy world model that leverages decoupled dynamic flow and image-assisted training strategy, substantially improving 4D scene forecasting performance. To simplify the training process, we discard the previous two-stage training strategy and innovatively reformulate the occupancy forecasting problem as a decoupled voxels warping process. Our model forecasts future dynamic voxels by warping existing observations using voxel flow, whereas static voxels are easily obtained through pose transformation. Moreover, our method incorporates an image-assisted training paradigm to enhance prediction reliability. Specifically, differentiable volume rendering is adopted to generate rendered depth maps through predicted future volumes, which are adopted in render-based photometric consistency. Experiments demonstrate the effectiveness of our approach, showcasing its state-of-the-art performance on the nuScenes and OpenScene benchmarks for 4D occupancy forecasting, end-to-end motion planning and point cloud forecasting. Concretely, it achieves state-of-the-art performances compared to existing 3D world models while incurring substantially lower computational costs.

Figures (4)

• Figure 1: Comparison with existing methods. (a) The pipeline of the existing method, i.e., OccWorld, adopts a two-stage training paradigm and autoregressive (AR) manner for inference. E and D denote the encoder, and decoder, respectively. (b) Our method utilizes an end-to-end and non-autoregressive (NAR) pipeline with the decoupled dynamic flow strategy for occupancy forecasting. The proposed image-assisted training enhances the performance.
• Figure 2: Overall pipeline of our method. It takes past images, occupancy, and ego poses as inputs, learning cross-modal information through a cross-modal spatial-temporal encoder. Then different decoders are responsible for predicting future frames in a non-autoregressive manner. And the proposed flow decoder and warping operation contained in the occupancy decoder could facilitate future occupancy forecasting. Additionally, an image-assisted training paradigm effectively imposes constraints on forecasted occupancy from the image domain, making more reliable occupancy predictions.
• Figure 3: Inner structure of SALT, warping and refinement module, and the image-assisted training. (a) The detailed structures of SALT, which replace the MLP and FFN in vanilla transformer with 2D convolutions and 3D convolutions respectively for capturing spatial-temporal dependencies. (b) We decouple the flow with the dynamic and static flow and warp the feature of the current frame for forecasting the future frame. The refinement module refines the coarse warping features. (c) The details of image-assisted training, where the Rendering-based Photometric Consistency (RPC) module is leveraged to further improve the forecasting performances with the depth map obtained by volume rendering.
• Figure 4: Qualitative results of 4D occupancy forecasting. Our method could foresee more reasonable drivable areas along with the semantic scene evolutions. However, the results generated by existing methods are inferior in the temporal consistency and dynamic object movements. Especially for the long-term predictions.