Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Benchmarking ground truth trajectories with robotic total stations

Effie Daum, Maxime Vaidis, François Pomerleau

TL;DR

Outdoor SLAM benchmarking requires accurate, reproducible ground-truth trajectories, but GNSS ground truth can drift over time. The paper compares robotic total stations (RTS) and GNSS using a standardized, repeatable protocol with careful calibration and processing to evaluate precision and reproducibility. Results show RTS achieving millimeter-scale accuracy and reproducibility (median around $6$–$8$ mm) across multiple deployments, outperforming GNSS, which exhibits higher dispersion (tens of millimeters to centimeters). The work provides a standardized RTS/GNSS benchmarking pipeline and metrics, suggesting that combining both systems yields robust six-DOF trajectory ground truth for fair SLAM evaluation in varied environments.

Abstract

Benchmarks stand as vital cornerstones in elevating SLAM algorithms within mobile robotics. Consequently, ensuring accurate and reproducible ground truth generation is vital for fair evaluation. A majority of outdoor ground truths are generated by GNSS, which can lead to discrepancies over time, especially in covered areas. However, research showed that RTS setups are more precise and can alternatively be used to generate these ground truths. In our work, we compare both RTS and GNSS systems' precision and repeatability through a set of experiments conducted weeks and months apart in the same area. We demonstrated that RTS setups give more reproducible results, with disparities having a median value of 8.6 mm compared to a median value of 10.6 cm coming from a GNSS setup. These results highlight that RTS can be considered to benchmark process for SLAM algorithms with higher precision.

Benchmarking ground truth trajectories with robotic total stations

TL;DR

Outdoor SLAM benchmarking requires accurate, reproducible ground-truth trajectories, but GNSS ground truth can drift over time. The paper compares robotic total stations (RTS) and GNSS using a standardized, repeatable protocol with careful calibration and processing to evaluate precision and reproducibility. Results show RTS achieving millimeter-scale accuracy and reproducibility (median around – mm) across multiple deployments, outperforming GNSS, which exhibits higher dispersion (tens of millimeters to centimeters). The work provides a standardized RTS/GNSS benchmarking pipeline and metrics, suggesting that combining both systems yields robust six-DOF trajectory ground truth for fair SLAM evaluation in varied environments.

Abstract

Benchmarks stand as vital cornerstones in elevating SLAM algorithms within mobile robotics. Consequently, ensuring accurate and reproducible ground truth generation is vital for fair evaluation. A majority of outdoor ground truths are generated by GNSS, which can lead to discrepancies over time, especially in covered areas. However, research showed that RTS setups are more precise and can alternatively be used to generate these ground truths. In our work, we compare both RTS and GNSS systems' precision and repeatability through a set of experiments conducted weeks and months apart in the same area. We demonstrated that RTS setups give more reproducible results, with disparities having a median value of 8.6 mm compared to a median value of 10.6 cm coming from a GNSS setup. These results highlight that RTS can be considered to benchmark process for SLAM algorithms with higher precision.
Paper Structure (6 sections, 2 figures)

This paper contains 6 sections, 2 figures.

Table of Contents

  1. Introduction
  2. Benchmarking Standardized Experiments
  3. Standardization of RTS and GNSS protocol
  4. Metrics
  5. Results
  6. Conclusion

Figures (2)

  • Figure 1: A RTS setup and GNSS antennas were used to record the trajectory of a Warthog Clearpath platform on the Université Laval campus. The color bar displays the average GNSS disparities obtained between two identical trajectories done at different times. The red sphere marks the location of our static RTK GNSS reference antenna.
  • Figure 2: Error resulting from (a) inter-prisms and inter-GNSS metrics and, (b) inter-experiments metrics presented in \ref{['subsec:metrics']}. The results from the RTS are depicted in blue, while those from the GNSS are represented in orange. The median error is displayed at the center of each box, and the Interquartile Range (IQR) is depicted on the side.