Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Gaussian Based Adaptive Multi-Modal 3D Semantic Occupancy Prediction

A. Enes Doruk

TL;DR

This work tackles the challenge of dense 3D semantic occupancy prediction for autonomous driving by introducing GaussianOcc3D, a memory-efficient Gaussian-based framework that fuses camera-derived semantics with LiDAR geometry. It replaces dense voxel queries with learnable 3D Gaussians and integrates four components—LDFA for robust 3D lifting, entropy-based feature smoothing to mitigate cross-modal noise, adaptive camera-LiDAR fusion for dynamic reliability weighting, and the Gauss-Mamba head based on Tri-Perspective View and linear-time state-space modeling—to decode global context efficiently. The approach achieves state-of-the-art results on OpenOccupancy and Occ3D-nuScenes (e.g., mIoU of 49.4%) and strong performance on SemanticKITTI, notably excelling in slender and irregular structures like vegetation, poles, and motorcycles. The combination of uncertainty-aware fusion, depth-aware LiDAR lifting, and an efficient Mamba-based decoder yields robustness under adverse weather (rain) and low-light conditions, with favorable computational efficiency compared to voxel- and transformer-heavy baselines. Overall, GaussianOcc3D advances practical, scalable multi-modal 3D occupancy perception with strong geometric fidelity and semantic richness, enabling safer autonomous navigation in complex environments.

Abstract

The sparse object detection paradigm shift towards dense 3D semantic occupancy prediction is necessary for dealing with long-tail safety challenges for autonomous vehicles. Nonetheless, the current voxelization methods commonly suffer from excessive computation complexity demands, where the fusion process is brittle, static, and breaks down under dynamic environmental settings. To this end, this research work enhances a novel Gaussian-based adaptive camera-LiDAR multimodal 3D occupancy prediction model that seamlessly bridges the semantic strengths of camera modality with the geometric strengths of LiDAR modality through a memory-efficient 3D Gaussian model. The proposed solution has four key components: (1) LiDAR Depth Feature Aggregation (LDFA), where depth-wise deformable sampling is employed for dealing with geometric sparsity, (2) Entropy-Based Feature Smoothing, where cross-entropy is employed for handling domain-specific noise, (3) Adaptive Camera-LiDAR Fusion, where dynamic recalibration of sensor outputs is performed based on model outputs, and (4) Gauss-Mamba Head that uses Selective State Space Models for global context decoding that enjoys linear computation complexity.

Gaussian Based Adaptive Multi-Modal 3D Semantic Occupancy Prediction

TL;DR

This work tackles the challenge of dense 3D semantic occupancy prediction for autonomous driving by introducing GaussianOcc3D, a memory-efficient Gaussian-based framework that fuses camera-derived semantics with LiDAR geometry. It replaces dense voxel queries with learnable 3D Gaussians and integrates four components—LDFA for robust 3D lifting, entropy-based feature smoothing to mitigate cross-modal noise, adaptive camera-LiDAR fusion for dynamic reliability weighting, and the Gauss-Mamba head based on Tri-Perspective View and linear-time state-space modeling—to decode global context efficiently. The approach achieves state-of-the-art results on OpenOccupancy and Occ3D-nuScenes (e.g., mIoU of 49.4%) and strong performance on SemanticKITTI, notably excelling in slender and irregular structures like vegetation, poles, and motorcycles. The combination of uncertainty-aware fusion, depth-aware LiDAR lifting, and an efficient Mamba-based decoder yields robustness under adverse weather (rain) and low-light conditions, with favorable computational efficiency compared to voxel- and transformer-heavy baselines. Overall, GaussianOcc3D advances practical, scalable multi-modal 3D occupancy perception with strong geometric fidelity and semantic richness, enabling safer autonomous navigation in complex environments.

Abstract

The sparse object detection paradigm shift towards dense 3D semantic occupancy prediction is necessary for dealing with long-tail safety challenges for autonomous vehicles. Nonetheless, the current voxelization methods commonly suffer from excessive computation complexity demands, where the fusion process is brittle, static, and breaks down under dynamic environmental settings. To this end, this research work enhances a novel Gaussian-based adaptive camera-LiDAR multimodal 3D occupancy prediction model that seamlessly bridges the semantic strengths of camera modality with the geometric strengths of LiDAR modality through a memory-efficient 3D Gaussian model. The proposed solution has four key components: (1) LiDAR Depth Feature Aggregation (LDFA), where depth-wise deformable sampling is employed for dealing with geometric sparsity, (2) Entropy-Based Feature Smoothing, where cross-entropy is employed for handling domain-specific noise, (3) Adaptive Camera-LiDAR Fusion, where dynamic recalibration of sensor outputs is performed based on model outputs, and (4) Gauss-Mamba Head that uses Selective State Space Models for global context decoding that enjoys linear computation complexity.
Paper Structure (55 sections, 30 equations, 15 figures, 9 tables)

This paper contains 55 sections, 30 equations, 15 figures, 9 tables.

Table of Contents

  1. Introduction
  2. The Paradigm Shift to 3D Occupancy
  3. Sensory Modalities and Limitations
  4. Multi-Modal Synergy: Representation and Fusion Strategies
  5. The Representation Crisis and the Gaussian Paradigm
  6. Problem Statement and Thesis Contributions
  7. Proposed Methodology
  8. LiDAR Depth Feature Aggregation
  9. Entropy-Based Feature Smoothing
  10. Adaptive Camera-LiDAR Fusion
  11. The Gauss-Mamba Head
  12. Summary of Contributions
  13. Chapter Organization
  14. Related Work
  15. Camera Based 3D Occupancy Prediction
  16. ...and 40 more sections

Figures (15)

  • Figure 1: Nuscenes sensor setup.
  • Figure 2: SemanticKITTI sensor setup.
  • Figure 3: Intersection Over Union (IoU).
  • Figure 4: Comparison of 12,800 and 25,600 Gaussian representations on the OpenOccupancy validation set.
  • Figure 5: Qualitative results on the OpenOccupancy validation set.
  • ...and 10 more figures