OccFormer: Dual-path Transformer for Vision-based 3D Semantic Occupancy Prediction

TL;DR

OccFormer tackles camera-based 3D semantic occupancy prediction by introducing a dual-path transformer encoder that splits 3D processing into local horizontal and global BEV pathways for long-range, dynamic reasoning. The decoder reinterprets Mask2Former for 3D occupancy with preserve-pooling and class-guided sampling to cope with sparsity and class imbalance, achieving state-of-the-art SSC on SemanticKITTI and competitive LiDAR segmentation on nuScenes with a single model. The approach demonstrates the effectiveness of dense 3D feature volumes and transformer-based 3D perception for autonomous driving, offering strong improvements over monocular and competing vision-based methods. The work provides practical insights into efficient 3D representation learning, multi-scale fusion, and robust handling of sparse supervision in 3D space.

Abstract

The vision-based perception for autonomous driving has undergone a transformation from the bird-eye-view (BEV) representations to the 3D semantic occupancy. Compared with the BEV planes, the 3D semantic occupancy further provides structural information along the vertical direction. This paper presents OccFormer, a dual-path transformer network to effectively process the 3D volume for semantic occupancy prediction. OccFormer achieves a long-range, dynamic, and efficient encoding of the camera-generated 3D voxel features. It is obtained by decomposing the heavy 3D processing into the local and global transformer pathways along the horizontal plane. For the occupancy decoder, we adapt the vanilla Mask2Former for 3D semantic occupancy by proposing preserve-pooling and class-guided sampling, which notably mitigate the sparsity and class imbalance. Experimental results demonstrate that OccFormer significantly outperforms existing methods for semantic scene completion on SemanticKITTI dataset and for LiDAR semantic segmentation on nuScenes dataset. Code is available at \url{https://github.com/zhangyp15/OccFormer}.

Figures (6)

• Figure 1: The framework of the proposed OccFormer for camera-based 3D semantic occupancy prediction. The pipeline consists of the image encoder for extracting multi-scale 2D features, the image-to-3D transformation for lifting the 2D features to 3D volumes, and the transformer-based encoder-decoder for obtaining 3D semantic features and predicting the 3D semantic occupancy.
• Figure 2: Illustration of the dual-path transformer block. The local path processes the 3D feature by applying the shared windowed attention to each horizontal slice, while the global path operates on the collapsed BEV feature for scene-level semantic layouts. The dual-path outputs are finally fused through the weighted summation. The skip connection is omitted.
• Figure 3: Qualitative results on SemanticKITTI validation set. The input monocular image is shown on the left and the 3D semantic occupancy results from MonoScene cao2022monoscene, our OccFormer, and the annotations are then visualized sequentially. The darker colors within the occupancy images represent the unseen parts out of the camera FOV.
• Figure 4: Qualitative results on nuScenes validation set. The leftmost column shows the input surrounding images, the following three columns visualize the LiDAR segmentation from TPVFormer tpvformer, our method, and the annotation. The final two columns visualize the predicted 3D semantic occupancy from TPVFormer and our method.
• Figure 5: Comparisons of the uniform sampling and the proposed class-guided sampling. The sample percent is computed as the average sample ratio with 10k times of sampling. The class-guided sampling can significantly improve the quality of supervision.