Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Real-time 3D semantic occupancy prediction for autonomous vehicles using memory-efficient sparse convolution

Samuel Sze, Lars Kunze

TL;DR

This work tackles real-time 3D semantic occupancy prediction for autonomous driving by fusing front-view camera and LiDAR information into a sparse, voxel-based representation and applying memory-efficient sparse convolutions via the Minkowski Engine. The authors jointly address 3D scene completion and semantic segmentation in a dual-UNet architecture, achieving competitive accuracy on Occ3D-nuScenes while maintaining real-time performance and low memory usage. Key contributions include a novel sparse 3D convolution model tailored for outdoor sparse scenes, a dense-to-sparse densification pipeline, and a class-balanced multi-task loss that improves per-voxel semantics. The results indicate practical benefits for real-time AV perception and suggest extensions to multi-view setups, distant-region completion, and self-supervised training to further enhance robustness.

Abstract

In autonomous vehicles, understanding the surrounding 3D environment of the ego vehicle in real-time is essential. A compact way to represent scenes while encoding geometric distances and semantic object information is via 3D semantic occupancy maps. State of the art 3D mapping methods leverage transformers with cross-attention mechanisms to elevate 2D vision-centric camera features into the 3D domain. However, these methods encounter significant challenges in real-time applications due to their high computational demands during inference. This limitation is particularly problematic in autonomous vehicles, where GPU resources must be shared with other tasks such as localization and planning. In this paper, we introduce an approach that extracts features from front-view 2D camera images and LiDAR scans, then employs a sparse convolution network (Minkowski Engine), for 3D semantic occupancy prediction. Given that outdoor scenes in autonomous driving scenarios are inherently sparse, the utilization of sparse convolution is particularly apt. By jointly solving the problems of 3D scene completion of sparse scenes and 3D semantic segmentation, we provide a more efficient learning framework suitable for real-time applications in autonomous vehicles. We also demonstrate competitive accuracy on the nuScenes dataset.

Real-time 3D semantic occupancy prediction for autonomous vehicles using memory-efficient sparse convolution

TL;DR

This work tackles real-time 3D semantic occupancy prediction for autonomous driving by fusing front-view camera and LiDAR information into a sparse, voxel-based representation and applying memory-efficient sparse convolutions via the Minkowski Engine. The authors jointly address 3D scene completion and semantic segmentation in a dual-UNet architecture, achieving competitive accuracy on Occ3D-nuScenes while maintaining real-time performance and low memory usage. Key contributions include a novel sparse 3D convolution model tailored for outdoor sparse scenes, a dense-to-sparse densification pipeline, and a class-balanced multi-task loss that improves per-voxel semantics. The results indicate practical benefits for real-time AV perception and suggest extensions to multi-view setups, distant-region completion, and self-supervised training to further enhance robustness.

Abstract

In autonomous vehicles, understanding the surrounding 3D environment of the ego vehicle in real-time is essential. A compact way to represent scenes while encoding geometric distances and semantic object information is via 3D semantic occupancy maps. State of the art 3D mapping methods leverage transformers with cross-attention mechanisms to elevate 2D vision-centric camera features into the 3D domain. However, these methods encounter significant challenges in real-time applications due to their high computational demands during inference. This limitation is particularly problematic in autonomous vehicles, where GPU resources must be shared with other tasks such as localization and planning. In this paper, we introduce an approach that extracts features from front-view 2D camera images and LiDAR scans, then employs a sparse convolution network (Minkowski Engine), for 3D semantic occupancy prediction. Given that outdoor scenes in autonomous driving scenarios are inherently sparse, the utilization of sparse convolution is particularly apt. By jointly solving the problems of 3D scene completion of sparse scenes and 3D semantic segmentation, we provide a more efficient learning framework suitable for real-time applications in autonomous vehicles. We also demonstrate competitive accuracy on the nuScenes dataset.
Paper Structure (18 sections, 3 equations, 7 figures, 6 tables)

This paper contains 18 sections, 3 equations, 7 figures, 6 tables.

Table of Contents

  1. INTRODUCTION
  2. Related Work
  3. Camera View Transformation
  4. 3D Semantic Scene Completion
  5. Sparse Convolution
  6. Problem Formulation
  7. Method
  8. System Pipeline
  9. Minkowski Engine
  10. Network Architecture
  11. Loss Function
  12. Experiments
  13. Experimental Setup
  14. Scene Completion
  15. Semantic Segmentation
  16. ...and 3 more sections

Figures (7)

  • Figure 1: Scene understanding methods using camera and LiDAR in nuScenes. Top two images shows perceived sensor output. Bottom two images shows scene predictions made from sensor output.
  • Figure 2: Overview of System Pipeline
  • Figure 3: Minkowski Engine Sparse Convolution Network Architecture. Scene Completion U-Net is shown in detail. MinkUNet is the Semantic Segmentation U-Net.
  • Figure 4: Qualitative predictions on Occ3D-nuScenes validation dataset. Semantic occupancy grid is viewed in 3rd person perspective angle. Each group of images consists of occupancy prediction image on the left, ground truth label on the right, and front camera image on top.
  • Figure 5: This example shows the model's ability to predict dynamic obstacles such as temporary road barriers installed for construction on drivable surface.
  • ...and 2 more figures