FastOcc: Accelerating 3D Occupancy Prediction by Fusing the 2D Bird's-Eye View and Perspective View

TL;DR

FastOcc tackles the real-time 3D occupancy prediction problem for autonomous driving by rethinking the 3D decoding path. It replaces expensive 3D convolutions with a lightweight 2D BEV decoding branch and uses interpolation to recover height information from image features, fused with interpolated voxel cues from multiple cameras. The method is depth-guided, BEV-supervised, and integrates BEV and voxel features to generate dense semantic occupancy maps efficiently. Empirical results on Occ3D-nuScenes show competitive accuracy with significantly lower latency, enhanced further by TensorRT acceleration, highlighting practical viability for real-time autonomous perception.

Abstract

In autonomous driving, 3D occupancy prediction outputs voxel-wise status and semantic labels for more comprehensive understandings of 3D scenes compared with traditional perception tasks, such as 3D object detection and bird's-eye view (BEV) semantic segmentation. Recent researchers have extensively explored various aspects of this task, including view transformation techniques, ground-truth label generation, and elaborate network design, aiming to achieve superior performance. However, the inference speed, crucial for running on an autonomous vehicle, is neglected. To this end, a new method, dubbed FastOcc, is proposed. By carefully analyzing the network effect and latency from four parts, including the input image resolution, image backbone, view transformation, and occupancy prediction head, it is found that the occupancy prediction head holds considerable potential for accelerating the model while keeping its accuracy. Targeted at improving this component, the time-consuming 3D convolution network is replaced with a novel residual-like architecture, where features are mainly digested by a lightweight 2D BEV convolution network and compensated by integrating the 3D voxel features interpolated from the original image features. Experiments on the Occ3D-nuScenes benchmark demonstrate that our FastOcc achieves state-of-the-art results with a fast inference speed.

Figures (5)

• Figure 1: Comparisons of the mIoU and runtime of various 3D occupancy prediction methods on the Occ3D-nuScenes tianOcc3DLargeScale3D2023 validation set.
• Figure 2: The pipeline of the proposed method. First, multi-camera features are extracted from image inputs with a backbone network. Then image features are transformed to the 3D space following the LSSphilionLiftSplatShoot2020 strategy. The voxel feature is collapsed to the BEV form and decoded in the 2D representation. Subsequently, the BEV features are upsampled, repeated, and supplemented with the voxel features interpolated from image features. BEV semantic segmentation is supervised as an auxiliary loss.
• Figure 3: The comparison of applying 2D FCN and 3D FCN. It is obvious that 2D FCN is highly efficient in terms of time and memory cost.
• Figure 4: In the upper dashed box, the volume grids are projected to multiple perspective images. Features of hit voxels on the sub-pixels are bilinearly interpolated and lifted to corresponding 3D space. Below, the absence of the $z$-axis of repeated BEV features can be completed by the interpolated features.
• Figure 5: Visualization of the occupancy prediction results on the validation set of Occ3D-nuScenestianOcc3DLargeScale3D2023.