H3O: Hyper-Efficient 3D Occupancy Prediction with Heterogeneous Supervision

TL;DR

H3O tackles the problem of efficient, vision-based 3D occupancy prediction by eliminating costly cross-attention in 2D-3D fusion and building a 3D voxel volume through perspective projection with per-voxel view averaging. It augments 3D occupancy learning with heterogeneous supervision from differentiable volume rendering of multi-view depth, semantics, and surface normals, supervised by 2D LiDAR labels and foundation-model outputs. The approach achieves state-of-the-art results on Occ3D-nuScenes and SemanticKITTI while demanding substantially less computation, thanks to its simple volume construction and per-pixel ray sampling strategy. The work demonstrates that rich auxiliary tasks can compensate for imperfect 3D ground truth and that efficiency-focused design enables deployment on resource-constrained platforms like autonomous vehicles.

Abstract

3D occupancy prediction has recently emerged as a new paradigm for holistic 3D scene understanding and provides valuable information for downstream planning in autonomous driving. Most existing methods, however, are computationally expensive, requiring costly attention-based 2D-3D transformation and 3D feature processing. In this paper, we present a novel 3D occupancy prediction approach, H3O, which features highly efficient architecture designs that incur a significantly lower computational cost as compared to the current state-of-the-art methods. In addition, to compensate for the ambiguity in ground-truth 3D occupancy labels, we advocate leveraging auxiliary tasks to complement the direct 3D supervision. In particular, we integrate multi-camera depth estimation, semantic segmentation, and surface normal estimation via differentiable volume rendering, supervised by corresponding 2D labels that introduces rich and heterogeneous supervision signals. We conduct extensive experiments on the Occ3D-nuScenes and SemanticKITTI benchmarks that demonstrate the superiority of our proposed H3O.

Figures (4)

• Figure 1: Accuracy (mIoU on Occ3D-nuScenes tian2024occ3d) vs. computation efficiency (Tera-MACs). Our proposed H3O achieves the best accuracy and efficiency when comparing to baseline and latest state-of-the-art methods, including BEVFormer li2022bevformer, TPVFormer huang2023tri, OccFormer zhang2023occformer, and RenderOcc pan2024renderocc. Note that in this work, we consider the setting where input multi-camera images are from the same time step, i.e., we do not consider video frames.
• Figure 2: Overview of our proposed H3O. First, an image backbone (e.g., ResNet he2016deep) extracts features from the multiple camera images. Then, a 3D grid is used to query the 2D image features, based on which H3O constructs a 3D feature volume. Specifically, we average features from different views and avoid costly operations like cross-attention, which leads to significantly better efficiency. The 3D feature volume is subsequently processed by 3D convolutions to generate occupancy predictions. During training, volume rendering is also performed based on the 3D feature volume to produce 2D predictions, including depth, semantics, and surface normals, which are supervised to provide additional training signals. Note that volume rendering is disabled at inference-time hence incurring zero extra cost.
• Figure 3: An example our multi-task supervision signals on nuScenes. Top row shows an RGB image and the normal map generated by Metric3Dv2 hu2024metric3d, which we use as ground-truth normal. Bottom row shows the depth and semantic labels projected from LiDAR points. Best viewed when zoomed in.
• Figure 4: Example qualitative 3D semantic occupancy prediction of H3O on Occ3D-nuScenes validation set. Cons. Veh stands for "Construction Vehicle" and Dri. Sur stands for "Drivable Surface". We see that H3O is able to capture objects with fine details at far distances and obscure locations, as well as under poor lighting conditions, e.g., far-away vehicles, tree in a low-constrast region of the top-right image. This validates the benefit of introducing rich supervision into occupancy learning. For the back camera (middle of bottom row) we visualize the ego-car body at the top, avoiding obscuring drivable surface. Best viewed in color and zoomed in.