DSOcc: Leveraging Depth Awareness and Semantic Aid to Boost Camera-Based 3D Semantic Occupancy Prediction

TL;DR

DSOcc tackles camera-based 3D semantic occupancy by introducing depth-aware soft occupancy and semantic-aided class inference. It embeds depth-derived occupancy confidence into image features and fuses multi-frame semantic segmentation with occupancy probabilities via a dual deformable-attention voxel fusion module. The approach achieves state-of-the-art mIoU among camera-based methods on SemanticKITTI and competitive results on SSCBench-KITTI-360 and Occ3D-nuScenes, with robustness to depth estimation noise and class imbalance. This work demonstrates practical potential for low-cost autonomous driving perception by improving both occupancy state and class prediction without expanding occupancy annotations.

Abstract

Camera-based 3D semantic occupancy prediction offers an efficient and cost-effective solution for perceiving surrounding scenes in autonomous driving. However, existing works rely on explicit occupancy state inference, leading to numerous incorrect feature assignments, and insufficient samples restrict the learning of occupancy class inference. To address these challenges, we propose leveraging \textbf{D}epth awareness and \textbf{S}emantic aid to boost camera-based 3D semantic \textbf{Occ}upancy prediction (\textbf{DSOcc}). We jointly perform occupancy state and occupancy class inference, where soft occupancy confidence is calculated by non-learning method and multiplied with image features to make voxels aware of depth, enabling adaptive implicit occupancy state inference. Instead of enhancing feature learning, we directly utilize well-trained image semantic segmentation and fuse multiple frames with their occupancy probabilities to aid occupancy class inference, thereby enhancing robustness. Experimental results demonstrate that DSOcc achieves state-of-the-art performance on the SemanticKITTI dataset among camera-based methods and achieves competitive performance on the SSCBench-KITTI-360 and Occ3D-nuScenes datasets. Code will be released on github.

Figures (8)

• Figure 1: Camera-based 3D semantic occupancy prediction methods typically rely on depth to determine occupancy states. Depth-threshold-based approaches, however, are prone to misclassifying occupied voxels as empty, which leads to accumulated errors in subsequent occupancy class inference. In contrast, we employ a soft occupancy confidence to implicitly represent the occupancy state, effectively addressing this issue. Furthermore, we introduce probability-fused image semantic segmentation to assist occupancy class inference, enabling robust and accurate reasoning without increasing the scale of occupancy annotations.
• Figure 2: The framework of DSOcc. To address the misclassification issue in occupancy state inference, we compute a soft occupancy confidence to implicitly represent the occupancy state and embed it into image features. To overcome the limitations of occupancy class reasoning, we fuse multi-frame semantic segmentation maps for better inference capacity and class distribution. Finally, a voxel fusion module is incorporated to integrate occupancy state and class predictions for the final output.
• Figure 3: Occupancy state inference of different methods. (a) ground truth. (b) without depth information. (c) explicit occupancy state inference using hard confidence truncation. (d) our depth-aware approach: embedding a soft truncation.
• Figure 4: The class distribution on SemanticKITTI validation dataset behley2019semantickitti. (a) the ground truth. (b) camera image semantic segmentation. (c) the prediction case when only use occupancy annotation during training phase. (d) projection shift between adjacent frames, A voxel may be projected onto different classes in multi-frame semantic segmentation maps.
• Figure 5: Voxel fusion module: The dual interaction mechanism aggregates segmentation and image feature information globally via deformable attention to obtain the fused voxel.