Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SparseOcc: Rethinking Sparse Latent Representation for Vision-Based Semantic Occupancy Prediction

Pin Tang, Zhongdao Wang, Guoqing Wang, Jilai Zheng, Xiangxuan Ren, Bailan Feng, Chao Ma

TL;DR

SparseOcc addresses the inefficiency of dense 3D latent representations in vision-based occupancy prediction by adopting a lossless sparse latent space. It introduces three components—a Sparse Latent Diffuser, a Sparse Feature Pyramid, and a Sparse Transformer Head—that operate exclusively on non-empty voxels, using kernel-decomposed 3D convolutions and sparse interpolation. The method achieves substantial FLOPs and memory reductions while improving semantic occupancy mIoU on nuScenes-Occupancy and matching or exceeding state-of-the-art baselines on SemanticKITTI. This sparse-geometry approach reduces hallucinations on empty voxels and provides a practical baseline for scalable 3D perception in autonomous driving.

Abstract

Vision-based perception for autonomous driving requires an explicit modeling of a 3D space, where 2D latent representations are mapped and subsequent 3D operators are applied. However, operating on dense latent spaces introduces a cubic time and space complexity, which limits scalability in terms of perception range or spatial resolution. Existing approaches compress the dense representation using projections like Bird's Eye View (BEV) or Tri-Perspective View (TPV). Although efficient, these projections result in information loss, especially for tasks like semantic occupancy prediction. To address this, we propose SparseOcc, an efficient occupancy network inspired by sparse point cloud processing. It utilizes a lossless sparse latent representation with three key innovations. Firstly, a 3D sparse diffuser performs latent completion using spatially decomposed 3D sparse convolutional kernels. Secondly, a feature pyramid and sparse interpolation enhance scales with information from others. Finally, the transformer head is redesigned as a sparse variant. SparseOcc achieves a remarkable 74.9% reduction on FLOPs over the dense baseline. Interestingly, it also improves accuracy, from 12.8% to 14.1% mIOU, which in part can be attributed to the sparse representation's ability to avoid hallucinations on empty voxels.

SparseOcc: Rethinking Sparse Latent Representation for Vision-Based Semantic Occupancy Prediction

TL;DR

SparseOcc addresses the inefficiency of dense 3D latent representations in vision-based occupancy prediction by adopting a lossless sparse latent space. It introduces three components—a Sparse Latent Diffuser, a Sparse Feature Pyramid, and a Sparse Transformer Head—that operate exclusively on non-empty voxels, using kernel-decomposed 3D convolutions and sparse interpolation. The method achieves substantial FLOPs and memory reductions while improving semantic occupancy mIoU on nuScenes-Occupancy and matching or exceeding state-of-the-art baselines on SemanticKITTI. This sparse-geometry approach reduces hallucinations on empty voxels and provides a practical baseline for scalable 3D perception in autonomous driving.

Abstract

Vision-based perception for autonomous driving requires an explicit modeling of a 3D space, where 2D latent representations are mapped and subsequent 3D operators are applied. However, operating on dense latent spaces introduces a cubic time and space complexity, which limits scalability in terms of perception range or spatial resolution. Existing approaches compress the dense representation using projections like Bird's Eye View (BEV) or Tri-Perspective View (TPV). Although efficient, these projections result in information loss, especially for tasks like semantic occupancy prediction. To address this, we propose SparseOcc, an efficient occupancy network inspired by sparse point cloud processing. It utilizes a lossless sparse latent representation with three key innovations. Firstly, a 3D sparse diffuser performs latent completion using spatially decomposed 3D sparse convolutional kernels. Secondly, a feature pyramid and sparse interpolation enhance scales with information from others. Finally, the transformer head is redesigned as a sparse variant. SparseOcc achieves a remarkable 74.9% reduction on FLOPs over the dense baseline. Interestingly, it also improves accuracy, from 12.8% to 14.1% mIOU, which in part can be attributed to the sparse representation's ability to avoid hallucinations on empty voxels.
Paper Structure (16 sections, 6 equations, 4 figures, 6 tables)

This paper contains 16 sections, 6 equations, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. 3D Scene Representation
  4. Scene Completion in Occupancy Prediction
  5. Proposed Approach
  6. Preliminary: Sparse Representation
  7. Sparse Latent Diffuser
  8. Sparse Feature Pyramid
  9. Sparse Transformer Head
  10. Objective Function
  11. Experiments
  12. Experimental Setup
  13. Benchmark Performance
  14. Qualitative Evaluation
  15. Ablation Studies
  16. ...and 1 more sections

Figures (4)

  • Figure 1: (a) Vision-based perception methods for autonomous driving typically first extract image features by a 2D latent encoder and then map them to 3D using view transformation. (b) For the 3D latent space, existing methods mostly employ the dense, BEV, or TPV representation, while we rethink the possibility of using sparse representation to achieve superior efficiency and accuracy.
  • Figure 2: Overview of the proposed approach. (a) Images captured by monocular or surrounding cameras are first passed to a 2D encoder, yielding 2D latent features. Then the latent features are mapped to 3D using the predicted depth map following the LSS lss. (b) SparseOcc adopts a sparse representation for the latent space. Upon this representation, we introduce three key building blocks: a latent diffuser that performs completion, a feature pyramid that enhances receptive filed, and a transformer head that predicts semantic occupancy.
  • Figure 3: Two building blocks of the sparse latent diffuser. (a) The sparse completion block diffuses non-empty features to empty neighbors, and (b) The contextual aggregation block aggregates geometry and semantic features without engaging in completion.
  • Figure 4: Qualitative results of 3D semantic occupancy on nuScenes-Occupancy validation set. The input multi-view images are shown on the leftmost and the occupancy predictions of C-CONet OpenOccupancy, our SparseOcc, and the ground-truth are then visualized sequentially. Compared to 3D dense representation based C-CONet yan20222dpass, our SparseOcc achieves better completion and segmentation as highlighted by the red circles.