SurroundOcc: Multi-Camera 3D Occupancy Prediction for Autonomous Driving

TL;DR

SurroundOcc tackles dense 3D scene understanding for autonomous driving using multi-camera inputs by predicting voxelized occupancy. It introduces a 2D-3D spatial attention mechanism to lift 2D image features into 3D volumes and employs a multi-scale 3D deconvolution framework with decayed supervisory signals to produce dense occupancy. A key contribution is a dense ground-truth generation pipeline that stitches multi-frame LiDAR data and densifies it with Poisson Reconstruction, enabling supervision without extensive new annotations. Experiments on nuScenes and SemanticKITTI demonstrate state-of-the-art occupancy prediction and competitive 3D reconstruction, highlighting the approach's potential for robust, occlusion-aware surround perception.

Abstract

3D scene understanding plays a vital role in vision-based autonomous driving. While most existing methods focus on 3D object detection, they have difficulty describing real-world objects of arbitrary shapes and infinite classes. Towards a more comprehensive perception of a 3D scene, in this paper, we propose a SurroundOcc method to predict the 3D occupancy with multi-camera images. We first extract multi-scale features for each image and adopt spatial 2D-3D attention to lift them to the 3D volume space. Then we apply 3D convolutions to progressively upsample the volume features and impose supervision on multiple levels. To obtain dense occupancy prediction, we design a pipeline to generate dense occupancy ground truth without expansive occupancy annotations. Specifically, we fuse multi-frame LiDAR scans of dynamic objects and static scenes separately. Then we adopt Poisson Reconstruction to fill the holes and voxelize the mesh to get dense occupancy labels. Extensive experiments on nuScenes and SemanticKITTI datasets demonstrate the superiority of our method. Code and dataset are available at https://github.com/weiyithu/SurroundOcc

Figures (8)

• Figure 1: The overview of SurroundOcc. Given multi-camera images, our method can predict volumetric occupancy of surrounding 3D scenes. To train the network, we design a pipeline to generate dense occupancy labels with sparse LiDAR points. Better viewed when zoomed in.
• Figure 2: The pipeline of the proposed method. First, we use a backbone to extract multi-scale features of multi-camera images. Then we adopt 2D-3D spatial attention to fuse multi-camera information and construct 3D volume features in a multi-scale fashion. Finally, the 3D deconvolution layer is used to upsample 3D volumes and occupancy prediction is supervised in each level.
• Figure 3: The comparison of 3D-based and BEV-based cross-view attention. The 3D-based attention can better preseve 3D information. For each 3D volume query, we project it to the corresponding 2D views to sample features.
• Figure 4: Dense occupancy ground truth generation. We first traverse all frames to stitch the multi-frame LiDAR points of dynamic objects and static scenes separately, and then merge them into a complete scene. Subsequently, we employ Poisson Reconstruction to densify the points and voxelize the resulting mesh to obtain a dense 3D occupancy. Finally, we use the Nearest Neighbor (NN) algorithm to assign semantic labels to dense voxels.
• Figure 5: Comparison on different occupancy labels. Compared with single-frame LiDAR points and the sparse occupancy converted from multi-frame points, our dense voxels are able to provide more realistic occupancy labels.