HybridOcc: NeRF Enhanced Transformer-based Multi-Camera 3D Occupancy Prediction

TL;DR

HybridOcc addresses occlusion-induced ambiguities in vision-based SSC by fusing a Transformer-based 2D-to-3D lifting module with a depth-supervised NeRF branch in a coarse-to-fine framework. The Transformer provides explicit, multi-scale occupancy cues, while the NeRF branch contributes depth supervision and occupancy inference for both visible and invisible voxels, guided by occupancy-aware ray sampling. The two branches generate complementary hybrid volume queries that are refined across scales, with a joint loss that balances explicit occupancy and implicit depth-based supervision. Experiments on nuScenes and SemanticKITTI show consistent gains over depth-prediction and standalone NeRF-based approaches, highlighting the practical impact for robust, end-to-end SSC in autonomous driving.

Abstract

Vision-based 3D semantic scene completion (SSC) describes autonomous driving scenes through 3D volume representations. However, the occlusion of invisible voxels by scene surfaces poses challenges to current SSC methods in hallucinating refined 3D geometry. This paper proposes HybridOcc, a hybrid 3D volume query proposal method generated by Transformer framework and NeRF representation and refined in a coarse-to-fine SSC prediction framework. HybridOcc aggregates contextual features through the Transformer paradigm based on hybrid query proposals while combining it with NeRF representation to obtain depth supervision. The Transformer branch contains multiple scales and uses spatial cross-attention for 2D to 3D transformation. The newly designed NeRF branch implicitly infers scene occupancy through volume rendering, including visible and invisible voxels, and explicitly captures scene depth rather than generating RGB color. Furthermore, we present an innovative occupancy-aware ray sampling method to orient the SSC task instead of focusing on the scene surface, further improving the overall performance. Extensive experiments on nuScenes and SemanticKITTI datasets demonstrate the effectiveness of our HybridOcc on the SSC task.

Figures (5)

• Figure 1: Comparison of depth-based methods, NeRF-based and our HybridOcc. (a) Advanced methods such as FB-Occ li2023fb and VoxFormer li2023voxformer require additional depth prediction networks for generating 3D voxels. (b) NeRF-based methods zhang2023occnerfpan2023renderocc only focus on the visible surface (blue sampling points) of the scene and render based on the transformed 3D voxel features. (c) In HybridOcc, the NeRF branch combined with the Transformer branch gradually refines SSC from coarse-to-fine. We propose a 3D occupancy-aware ray sampling (yellow sampling points) to enable the model to focus on occupied voxels of all scenes rather than visible surfaces.
• Figure 2: Illustration of the 3D occupancy prediction data set. SurroundOcc-nuScenes wei2023surroundocc and SemanticKITT behley2019semantickitti evaluate all occupied voxels, while Occ3D-nuScenes and Occ3D-Waymo tian2024occ3d only evaluate visible surfaces.
• Figure 3: The pipeline of the proposed HybridOcc for multi-camera 3D semantic occupancy prediction. It consists of the image backbone for extracting multi-scale features and the dual branch composed of Transformer and NeRF to learn a sparse 3D feature volume from coarse to fine. The Transformer branch contains a 2D to 3D transformation module for lifting the 2D features to 3D volumes, and the NeRF branch obtains supervision from depth signals to enhance the Transformer branch.
• Figure 4: Comparisons of the proposed occupancy-aware ray sampling with hierarchical volume sampling. Our ray sampling strategy focuses on sampling all visible or invisible occupied voxels passing through the ray.
• Figure 5: Visualizations on nuScenes validation set. The leftmost column is the input multi-camera surround image, and the next four columns show the ground truth semantic occupancy, the 3D semantic occupancy predicted by our HybridOcc, OccFormer Zhang2023OccFormerDT and SurroundOcc wei2023surroundocc.