Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

3DRef: 3D Dataset and Benchmark for Reflection Detection in RGB and Lidar Data

Xiting Zhao, Sören Schwertfeger

TL;DR

The paper addresses the persistent challenge of reflective surfaces in 3D mapping by introducing 3DRef, the first large-scale multi-modal dataset with 51800+ aligned RGB and multi-return LiDAR samples, textured ground-truth meshes, and 2D/3D semantic labels for glass, mirrors, and other reflections. It provides automatic point-cloud labeling via textured meshes and ray-casting, and establishes benchmarks for both LiDAR-based and RGB-based reflection detection methods, analyzing the role of multi-return information. Key contributions include the dataset ground truth pipeline, comprehensive statistics across three indoor sequences, and extensive benchmark results showing significant performance gains when retraining models on 3DRef. The dataset is publicly available and poised to advance robust reflection detection and reliable 3D mapping in real-world applications through improved cross-modal understanding and future sensor fusion research.

Abstract

Reflective surfaces present a persistent challenge for reliable 3D mapping and perception in robotics and autonomous systems. However, existing reflection datasets and benchmarks remain limited to sparse 2D data. This paper introduces the first large-scale 3D reflection detection dataset containing more than 50,000 aligned samples of multi-return Lidar, RGB images, and 2D/3D semantic labels across diverse indoor environments with various reflections. Textured 3D ground truth meshes enable automatic point cloud labeling to provide precise ground truth annotations. Detailed benchmarks evaluate three Lidar point cloud segmentation methods, as well as current state-of-the-art image segmentation networks for glass and mirror detection. The proposed dataset advances reflection detection by providing a comprehensive testbed with precise global alignment, multi-modal data, and diverse reflective objects and materials. It will drive future research towards reliable reflection detection. The dataset is publicly available at http://3dref.github.io

3DRef: 3D Dataset and Benchmark for Reflection Detection in RGB and Lidar Data

TL;DR

The paper addresses the persistent challenge of reflective surfaces in 3D mapping by introducing 3DRef, the first large-scale multi-modal dataset with 51800+ aligned RGB and multi-return LiDAR samples, textured ground-truth meshes, and 2D/3D semantic labels for glass, mirrors, and other reflections. It provides automatic point-cloud labeling via textured meshes and ray-casting, and establishes benchmarks for both LiDAR-based and RGB-based reflection detection methods, analyzing the role of multi-return information. Key contributions include the dataset ground truth pipeline, comprehensive statistics across three indoor sequences, and extensive benchmark results showing significant performance gains when retraining models on 3DRef. The dataset is publicly available and poised to advance robust reflection detection and reliable 3D mapping in real-world applications through improved cross-modal understanding and future sensor fusion research.

Abstract

Reflective surfaces present a persistent challenge for reliable 3D mapping and perception in robotics and autonomous systems. However, existing reflection datasets and benchmarks remain limited to sparse 2D data. This paper introduces the first large-scale 3D reflection detection dataset containing more than 50,000 aligned samples of multi-return Lidar, RGB images, and 2D/3D semantic labels across diverse indoor environments with various reflections. Textured 3D ground truth meshes enable automatic point cloud labeling to provide precise ground truth annotations. Detailed benchmarks evaluate three Lidar point cloud segmentation methods, as well as current state-of-the-art image segmentation networks for glass and mirror detection. The proposed dataset advances reflection detection by providing a comprehensive testbed with precise global alignment, multi-modal data, and diverse reflective objects and materials. It will drive future research towards reliable reflection detection. The dataset is publicly available at http://3dref.github.io
Paper Structure (25 sections, 7 figures, 6 tables)

This paper contains 25 sections, 7 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Reflection Datasets
  4. Reflection Detection Methods
  5. Reflection Detection Methods Using 3D Lidar
  6. Reflection Detection Methods Using RGB Cameras
  7. Other Reflection Detection Methods
  8. Data Collection Platform
  9. Polar 3D Scanner with Ouster Lidar and Insta360 camera
  10. Livox Avix Triple Return Lidar
  11. Pandar Qt64 Dual Return Lidar
  12. Time Sync and Calibration
  13. Dataset
  14. Data Collection Process
  15. Data Annotation and Labeling
  16. ...and 10 more sections

Figures (7)

  • Figure 1: The 3DRef dataset uses a labeled mesh to provide multi-modal reflection detection data, including three different multi-return Lidar and RGB image with mask. The label across all reflective material including glass, mirror and other reflective objects.
  • Figure 2: Data Collection Platform
  • Figure 3: Other Reflective Objects
  • Figure 4: Sequence 1: Corridor with Mirrors
  • Figure 5: Sequence 2: Rooms
  • ...and 2 more figures