Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GaussianOcc3D: A Gaussian-Based Adaptive Multi-modal 3D Occupancy Prediction

A. Enes Doruk, Hasan F. Ates

TL;DR

GaussianOcc3D addresses dense 3D semantic occupancy prediction by fusing camera semantics and LiDAR geometry within a memory-efficient shared Gaussian space. It introduces four modules—LDFA for LiDAR lifting, EBFS for cross-modal smoothing, ACLF for adaptive fusion, and Gauss-Mamba Head for global context—to jointly learn Gaussian primitives and predict a voxelized occupancy grid. The approach achieves state-of-the-art mIoU on Occ3D (49.4%), SurroundOcc (28.9%), and SemanticKITTI (25.2%), while maintaining robustness under rain and nighttime conditions. By operating on continuous Gaussian primitives rather than voxel grids, GaussianOcc3D reduces computational burden and preserves high-fidelity geometry and semantics, enabling reliable 3D scene understanding for autonomous driving.

Abstract

3D semantic occupancy prediction is a pivotal task in autonomous driving, providing a dense and fine-grained understanding of the surrounding environment, yet single-modality methods face trade-offs between camera semantics and LiDAR geometry. Existing multi-modal frameworks often struggle with modality heterogeneity, spatial misalignment, and the representation crisis--where voxels are computationally heavy and BEV alternatives are lossy. We present GaussianOcc3D, a multi-modal framework bridging camera and LiDAR through a memory-efficient, continuous 3D Gaussian representation. We introduce four modules: (1) LiDAR Depth Feature Aggregation (LDFA), using depth-wise deformable sampling to lift sparse signals onto Gaussian primitives; (2) Entropy-Based Feature Smoothing (EBFS) to mitigate domain noise; (3) Adaptive Camera-LiDAR Fusion (ACLF) with uncertainty-aware reweighting for sensor reliability; and (4) a Gauss-Mamba Head leveraging Selective State Space Models for global context with linear complexity. Evaluations on Occ3D, SurroundOcc, and SemanticKITTI benchmarks demonstrate state-of-the-art performance, achieving mIoU scores of 49.4%, 28.9%, and 25.2% respectively. GaussianOcc3D exhibits superior robustness across challenging rainy and nighttime conditions.

GaussianOcc3D: A Gaussian-Based Adaptive Multi-modal 3D Occupancy Prediction

TL;DR

GaussianOcc3D addresses dense 3D semantic occupancy prediction by fusing camera semantics and LiDAR geometry within a memory-efficient shared Gaussian space. It introduces four modules—LDFA for LiDAR lifting, EBFS for cross-modal smoothing, ACLF for adaptive fusion, and Gauss-Mamba Head for global context—to jointly learn Gaussian primitives and predict a voxelized occupancy grid. The approach achieves state-of-the-art mIoU on Occ3D (49.4%), SurroundOcc (28.9%), and SemanticKITTI (25.2%), while maintaining robustness under rain and nighttime conditions. By operating on continuous Gaussian primitives rather than voxel grids, GaussianOcc3D reduces computational burden and preserves high-fidelity geometry and semantics, enabling reliable 3D scene understanding for autonomous driving.

Abstract

3D semantic occupancy prediction is a pivotal task in autonomous driving, providing a dense and fine-grained understanding of the surrounding environment, yet single-modality methods face trade-offs between camera semantics and LiDAR geometry. Existing multi-modal frameworks often struggle with modality heterogeneity, spatial misalignment, and the representation crisis--where voxels are computationally heavy and BEV alternatives are lossy. We present GaussianOcc3D, a multi-modal framework bridging camera and LiDAR through a memory-efficient, continuous 3D Gaussian representation. We introduce four modules: (1) LiDAR Depth Feature Aggregation (LDFA), using depth-wise deformable sampling to lift sparse signals onto Gaussian primitives; (2) Entropy-Based Feature Smoothing (EBFS) to mitigate domain noise; (3) Adaptive Camera-LiDAR Fusion (ACLF) with uncertainty-aware reweighting for sensor reliability; and (4) a Gauss-Mamba Head leveraging Selective State Space Models for global context with linear complexity. Evaluations on Occ3D, SurroundOcc, and SemanticKITTI benchmarks demonstrate state-of-the-art performance, achieving mIoU scores of 49.4%, 28.9%, and 25.2% respectively. GaussianOcc3D exhibits superior robustness across challenging rainy and nighttime conditions.
Paper Structure (13 sections, 12 equations, 5 figures, 8 tables)

This paper contains 13 sections, 12 equations, 5 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Framework Overview
  5. Multi-Modal Feature Aggregation
  6. Entropy-Based Feature Smoothing
  7. Adaptive Camera-LiDAR Fusion
  8. Gauss-Mamba Head
  9. Experiments
  10. Experimental Setup
  11. Main Results
  12. Ablation Studies
  13. Conclusion

Figures (5)

  • Figure 1: The overall architecture of the proposed GaussianOcc3D framework.
  • Figure 2: Architecture of the LiDAR Depth Feature Aggregation Module.
  • Figure 3: Architecture of the Adaptive Camera-LiDAR Fusion Module.
  • Figure 4: Comparison of $12,800$ and $25,600$ Gaussian representations on the OpenOccupancy validation set.
  • Figure 5: Qualitative result on the SurroundOcc surroundocc validation set.