OccGen: Generative Multi-modal 3D Occupancy Prediction for Autonomous Driving

TL;DR

OccGen introduces a generative diffusion-based framework for 3D semantic occupancy prediction that progresses from noisy 3D voxels to detailed occupancy maps conditioned on multi-modal inputs. By separating a single-pass encoder from a multi-step progressive decoder, OccGen achieves coarse-to-fine reconstruction with the ability to quantify uncertainty through incremental denoising steps. The approach delivers state-of-the-art results on nuScenes-Occupancy and SemanticKITTI, with clear gains in mIoU across camera-only, LiDAR-only, and multi-modal setups, and demonstrates meaningful qualitative improvements in scene completion and consistency. The work highlights the practical benefits of progressive inference, uncertainty estimation, and the potential for balancing computation against accuracy in autonomous driving perception systems.

Abstract

Existing solutions for 3D semantic occupancy prediction typically treat the task as a one-shot 3D voxel-wise segmentation perception problem. These discriminative methods focus on learning the mapping between the inputs and occupancy map in a single step, lacking the ability to gradually refine the occupancy map and the reasonable scene imaginative capacity to complete the local regions somewhere. In this paper, we introduce OccGen, a simple yet powerful generative perception model for the task of 3D semantic occupancy prediction. OccGen adopts a ''noise-to-occupancy'' generative paradigm, progressively inferring and refining the occupancy map by predicting and eliminating noise originating from a random Gaussian distribution. OccGen consists of two main components: a conditional encoder that is capable of processing multi-modal inputs, and a progressive refinement decoder that applies diffusion denoising using the multi-modal features as conditions. A key insight of this generative pipeline is that the diffusion denoising process is naturally able to model the coarse-to-fine refinement of the dense 3D occupancy map, therefore producing more detailed predictions. Extensive experiments on several occupancy benchmarks demonstrate the effectiveness of the proposed method compared to the state-of-the-art methods. For instance, OccGen relatively enhances the mIoU by 9.5%, 6.3%, and 13.3% on nuScenes-Occupancy dataset under the muli-modal, LiDAR-only, and camera-only settings, respectively. Moreover, as a generative perception model, OccGen exhibits desirable properties that discriminative models cannot achieve, such as providing uncertainty estimates alongside its multiple-step predictions.

Figures (10)

• Figure 1: (a) The generative diagram of semantic segmentation (seg.), object detection (det.), and 3D semantic occupancy prediction (occ.). (b) Compared to previous discriminative methods with a single forward evaluation scheme, our OccGen is a generative model that can generate occupancy map in a coarse-to-fine manner.
• Figure 2: The proposed OccGen framework. It has an encoder-decoder structure. The conditional encoder extracts the features from the inputs as the condition. The progressive refinement decoder consists of a stack of refinement layers and an occupancy head, which takes the 3D noise map, sampling step, and conditional multi-scale fusion features as inputs and progressively generates the occupancy prediction.
• Figure 3: The detailed architectures of (a) multi-modal encoder and (b) refinement layer. The multi-modal encoder is a two-stream structure, comprising LiDAR and camera streams. The refinement layer consists of three main components, i.e., 3D deformable cross-attention, self-attention, and time diffusion modules.
• Figure 4: Qualitative results of the 3D semantic occupancy predictions on nuScenes-Occupancy. The leftmost column shows the input surrounding images, and the following four columns visualize the 3D semantic occupancy results from the ground truth, CONetwang2023openoccupancy, OccGen(step1), and OccGen(step2). The regions highlighted by rectangles indicate that these areas have obvious differences (better viewed when zoomed in).
• Figure 5: The results of multiple inferences on nuScenese-Occupancy under multi-modal setting.