Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

OccupancyDETR: Using DETR for Mixed Dense-sparse 3D Occupancy Prediction

Yupeng Jia, Jie He, Runze Chen, Fang Zhao, Haiyong Luo

TL;DR

OccupancyDETR addresses monocular 3D semantic occupancy by introducing a DETR-like object detection module that provides priors for a mixed dense-sparse 3D occupancy decoder. Foreground objects are decoded densely while background objects use sparse decoding, with MaskFormer used for background semantics. To mitigate slow convergence of DETR-like models, the authors propose an early matching pretraining strategy. Experiments on SemanticKITTI show real-time performance with improved handling of small objects and competitive accuracy at a reduced resource footprint. The work presents a practical, scalable approach to real-time 3D occupancy perception from monocular input and suggests easy extension to multi-frame/multi-camera setups.

Abstract

Visual-based 3D semantic occupancy perception is a key technology for robotics, including autonomous vehicles, offering an enhanced understanding of the environment by 3D. This approach, however, typically requires more computational resources than BEV or 2D methods. We propose a novel 3D semantic occupancy perception method, OccupancyDETR, which utilizes a DETR-like object detection, a mixed dense-sparse 3D occupancy decoder. Our approach distinguishes between foreground and background within a scene. Initially, foreground objects are detected using the DETR-like object detection. Subsequently, queries for both foreground and background objects are fed into the mixed dense-sparse 3D occupancy decoder, performing upsampling in dense and sparse methods, respectively. Finally, a MaskFormer is utilized to infer the semantics of the background voxels. Our approach strikes a balance between efficiency and accuracy, achieving faster inference times, lower resource consumption, and improved performance for small object detection. We demonstrate the effectiveness of our proposed method on the SemanticKITTI dataset, showcasing an mIoU of 14 and a processing speed of 10 FPS, thereby presenting a promising solution for real-time 3D semantic occupancy perception.

OccupancyDETR: Using DETR for Mixed Dense-sparse 3D Occupancy Prediction

TL;DR

OccupancyDETR addresses monocular 3D semantic occupancy by introducing a DETR-like object detection module that provides priors for a mixed dense-sparse 3D occupancy decoder. Foreground objects are decoded densely while background objects use sparse decoding, with MaskFormer used for background semantics. To mitigate slow convergence of DETR-like models, the authors propose an early matching pretraining strategy. Experiments on SemanticKITTI show real-time performance with improved handling of small objects and competitive accuracy at a reduced resource footprint. The work presents a practical, scalable approach to real-time 3D occupancy perception from monocular input and suggests easy extension to multi-frame/multi-camera setups.

Abstract

Visual-based 3D semantic occupancy perception is a key technology for robotics, including autonomous vehicles, offering an enhanced understanding of the environment by 3D. This approach, however, typically requires more computational resources than BEV or 2D methods. We propose a novel 3D semantic occupancy perception method, OccupancyDETR, which utilizes a DETR-like object detection, a mixed dense-sparse 3D occupancy decoder. Our approach distinguishes between foreground and background within a scene. Initially, foreground objects are detected using the DETR-like object detection. Subsequently, queries for both foreground and background objects are fed into the mixed dense-sparse 3D occupancy decoder, performing upsampling in dense and sparse methods, respectively. Finally, a MaskFormer is utilized to infer the semantics of the background voxels. Our approach strikes a balance between efficiency and accuracy, achieving faster inference times, lower resource consumption, and improved performance for small object detection. We demonstrate the effectiveness of our proposed method on the SemanticKITTI dataset, showcasing an mIoU of 14 and a processing speed of 10 FPS, thereby presenting a promising solution for real-time 3D semantic occupancy perception.
Paper Structure (16 sections, 1 equation, 7 figures, 2 tables)

This paper contains 16 sections, 1 equation, 7 figures, 2 tables.

Table of Contents

  1. INTRODUCTION
  2. RELATED WORK
  3. Camera-based BEV Perception
  4. 3D Semantic Occupancy Prediction
  5. Mask classification-based method
  6. APPROACH
  7. Overview
  8. Object Detection Module
  9. Mixed Dense-sparse 3D Occupancy Decoder
  10. Training strategy
  11. EXPERIMENTS
  12. Datasets
  13. Experiments Setup
  14. Main Results
  15. Comparative Study on Early Matching Pretraining
  16. ...and 1 more sections

Figures (7)

  • Figure 1: Concept diagram of 3D semantic occupancy perception inspired by human high-level visual processing.
  • Figure 2: The system diagram of OccupancyDETR. Arrows indicate how data flow within the system.
  • Figure 3: Data flow diagram of the object detection module, with yellow and blue arrows representing the data flows in training and early matching pretraining respectively.
  • Figure 4: The detailed architecture of mixed dense-sparse occupancy decoder
  • Figure 5: Qualitative results on SemanticKITTI test set.
  • ...and 2 more figures