SUG-Occ: An Explicit Semantics and Uncertainty Guided Sparse Learning Framework for Real-Time 3D Occupancy Prediction

TL;DR

The paper tackles real-time 3D semantic occupancy prediction for autonomous driving, addressing the prohibitive cost of dense 3D volumes. It introduces SUG-Occ, a three-stage framework that combines semantics and uncertainty priors with an explicit distance encoding, a cascade sparse completion network using hyper-cross sparse convolutions and generative upsampling, and an OCR-based mask decoder to refine voxel predictions with compact context. The approach yields state-of-the-art mean IoU and substantial efficiency gains on SemanticKITTI, demonstrating the practicality of real-time, voxel-level scene understanding. This work advances scalable 3D perception by merging principled priors, sparse computation, and context-aware decoding for safer, faster autonomous driving systems.

Abstract

As autonomous driving moves toward full scene understanding, 3D semantic occupancy prediction has emerged as a crucial perception task, offering voxel-level semantics beyond traditional detection and segmentation paradigms. However, such a refined representation for scene understanding incurs prohibitive computation and memory overhead, posing a major barrier to practical real-time deployment. To address this, we propose SUG-Occ, an explicit Semantics and Uncertainty Guided Sparse Learning Enabled 3D Occupancy Prediction Framework, which exploits the inherent sparsity of 3D scenes to reduce redundant computation while maintaining geometric and semantic completeness. Specifically, we first utilize semantic and uncertainty priors to suppress projections from free space during view transformation while employing an explicit unsigned distance encoding to enhance geometric consistency, producing a structurally consistent sparse 3D representation. Secondly, we design an cascade sparse completion module via hyper cross sparse convolution and generative upsampling to enable efficiently coarse-to-fine reasoning. Finally, we devise an object contextual representation (OCR) based mask decoder that aggregates global semantic context from sparse features and refines voxel-wise predictions via lightweight query-context interactions, avoiding expensive attention operations over volumetric features. Extensive experiments on SemanticKITTI benchmark demonstrate that the proposed approach outperforms the baselines, achieving a 7.34/% improvement in accuracy and a 57.8\% gain in efficiency.

Figures (6)

• Figure 1: Illustration of the inherent sparsity of 3D semantic occupancy prediction.
• Figure 2: Overview of the proposed SUG-Occ framework. At first, image features extracted from a single-frame camera input are selectively lifted into 3D space using semantic and depth uncertainty priors, yielding a sparse and structurally coherent initialization with explicit distance encoding. Then, an efficient cascade sparse completion network progressively reconstructs geometry and semantics while tightly controlling computational cost through generative upsampling and soft pruning. Finally, an OCR-based mask decoder further refines predictions by restricting attention to compact object contextual representations.
• Figure 3: Illustration of the multi-scale convolutional attention, which is consist of multi-scale decomposed depth-wise convolution for context aggregation, a point-wise convolution for channel mixing and a convolutional attention for residual operation.
• Figure 4: Illustration of the proposed hyper cross residual block. Compared with the conventional residual block using two stacked $3\times3\times3$ sparse convolutions, hyper cross residual block apply a three layers $3{+}2{+}2$ hyper cross convolution stack.
• Figure 5: Visualization of the receptive field expansion achieved by stacking hyper cross convolutions. As the number of layers increases, the effective receptive field progressively extends from axial neighbors to diagonal spatial locations.