Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Hilti SLAM Challenge 2023: Benchmarking Single + Multi-session SLAM across Sensor Constellations in Construction

Ashish Devadas Nair, Julien Kindle, Plamen Levchev, Davide Scaramuzza

TL;DR

The Hilti SLAM Challenge 2023 paper presents a publicly available dataset and benchmark to evaluate single- and multi-session SLAM across multiple sensor configurations in construction environments. It introduces a robot-mounted sensor suite and a novel LiDAR-observable GCP with mm-level position estimation, enabling accurate Absolute Trajectory Error evaluation in cross-device scenarios. The benchmark extends to multi-session tracking, updated scoring brackets, calibration options, and privacy safeguards, with top systems achieving sub-centimeter to centimeter ATE in lidar and improving parity with vision-based SLAM. Overall, the work demonstrates improved participation, robust single-session performance across sensor constellations, and emerging multi-session capabilities, signaling practical impact for construction robotics and autonomous systems.

Abstract

Simultaneous Localization and Mapping systems are a key enabler for positioning in both handheld and robotic applications. The Hilti SLAM Challenges organized over the past years have been successful at benchmarking some of the world's best SLAM Systems with high accuracy. However, more capabilities of these systems are yet to be explored, such as platform agnosticism across varying sensor suites and multi-session SLAM. These factors indirectly serve as an indicator of robustness and ease of deployment in real-world applications. There exists no dataset plus benchmark combination publicly available, which considers these factors combined. The Hilti SLAM Challenge 2023 Dataset and Benchmark addresses this issue. Additionally, we propose a novel fiducial marker design for a pre-surveyed point on the ground to be observable from an off-the-shelf LiDAR mounted on a robot, and an algorithm to estimate its position at mm-level accuracy. Results from the challenge show an increase in overall participation, single-session SLAM systems getting increasingly accurate, successfully operating across varying sensor suites, but relatively few participants performing multi-session SLAM. Dataset URL: https://www.hilti-challenge.com/dataset-2023.html

Hilti SLAM Challenge 2023: Benchmarking Single + Multi-session SLAM across Sensor Constellations in Construction

TL;DR

The Hilti SLAM Challenge 2023 paper presents a publicly available dataset and benchmark to evaluate single- and multi-session SLAM across multiple sensor configurations in construction environments. It introduces a robot-mounted sensor suite and a novel LiDAR-observable GCP with mm-level position estimation, enabling accurate Absolute Trajectory Error evaluation in cross-device scenarios. The benchmark extends to multi-session tracking, updated scoring brackets, calibration options, and privacy safeguards, with top systems achieving sub-centimeter to centimeter ATE in lidar and improving parity with vision-based SLAM. Overall, the work demonstrates improved participation, robust single-session performance across sensor constellations, and emerging multi-session capabilities, signaling practical impact for construction robotics and autonomous systems.

Abstract

Simultaneous Localization and Mapping systems are a key enabler for positioning in both handheld and robotic applications. The Hilti SLAM Challenges organized over the past years have been successful at benchmarking some of the world's best SLAM Systems with high accuracy. However, more capabilities of these systems are yet to be explored, such as platform agnosticism across varying sensor suites and multi-session SLAM. These factors indirectly serve as an indicator of robustness and ease of deployment in real-world applications. There exists no dataset plus benchmark combination publicly available, which considers these factors combined. The Hilti SLAM Challenge 2023 Dataset and Benchmark addresses this issue. Additionally, we propose a novel fiducial marker design for a pre-surveyed point on the ground to be observable from an off-the-shelf LiDAR mounted on a robot, and an algorithm to estimate its position at mm-level accuracy. Results from the challenge show an increase in overall participation, single-session SLAM systems getting increasingly accurate, successfully operating across varying sensor suites, but relatively few participants performing multi-session SLAM. Dataset URL: https://www.hilti-challenge.com/dataset-2023.html
Paper Structure (28 sections, 4 equations, 5 figures, 2 tables, 1 algorithm)

This paper contains 28 sections, 4 equations, 5 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Hardware
  4. IMU Calibration
  5. Stereo-Camera Calibration
  6. Extrinsic Calibration
  7. Registering of Ground Control Points
  8. Dataset
  9. Location and Data Overview
  10. Challenge Sequences
  11. Site 1 [Easy-Medium]
  12. Site 2 [Easy-Hard]
  13. Site 3 [Easy-Hard]
  14. Additional Sequences
  15. Ground Truth Extraction
  16. ...and 13 more sections

Figures (5)

  • Figure 1: Our SLAM data acquisition devices (a,b); and Ground Truth extraction approach (c).
  • Figure 2: Trailblazer's sensor suite (a) and calibration data collection procedure (b).
  • Figure 3: GCP detector visualizations (a,b) and test setup (c).
  • Figure 4: GCP Estimation Accuracy Evaluation. Left: GCP Estimation Relative Error box plots. Right: Rayleigh Distribution of 3DoF Euclidean Error.
  • Figure 5: Dataset Characteristics