Quantitative 3D Map Accuracy Evaluation Hardware and Algorithm for LiDAR(-Inertial) SLAM

TL;DR

The paper tackles the problem of quantitatively evaluating 3D maps produced by LiDAR(-Inertial) SLAM in outdoor environments. It introduces a robust outdoor LiDAR target and an automatic pose-extraction pipeline using $K$-means clustering, RANSAC, and SVD, synchronized with GPS ground truth. It defines two error metrics, $E_{rel}$ and $E_{abs}$, to capture global and local map accuracy, and validates them with five sequences from a highway construction site, showing that larger map extents can inflate $E_{rel}$ while $E_{abs}$ provides location-specific accuracy. It provides open-source implementations and advances repeatable, user-independent map accuracy assessment for LiDAR SLAM systems.

Abstract

Accuracy evaluation of a 3D pointcloud map is crucial for the development of autonomous driving systems. In this work, we propose a user-independent software/hardware system that can quantitatively evaluate the accuracy of a 3D pointcloud map acquired from LiDAR(-Inertial) SLAM. We introduce a LiDAR target that functions robustly in the outdoor environment, while remaining observable by LiDAR. We also propose a software algorithm that automatically extracts representative points and calculates the accuracy of the 3D pointcloud map by leveraging GPS position data. This methodology overcomes the limitations of the manual selection method, that its result varies between users. Furthermore, two different error metrics, relative and absolute errors, are introduced to analyze the accuracy from different perspectives. Our implementations are available at: https://github.com/SangwooJung98/3D_Map_Evaluation

Figures (6)

• Figure 1: An overview of the absolute and relative error metrics. Absolute error is derived from the distance between $N$ pairs of corresponding target pose and ground truth pose. Relative error is derived from the distance between every two target poses.
• Figure 2: Pipeline of the algorithm. The target generation(green) process can be performed with any LiDAR based SLAM algorithms. Using the target pointclouds and GPS poses, target poses on the map are estimated (blue). Map accuracy is evaluated using the GPS target pose and estimated target pose (red).
• Figure 3: \ref{['fig:target1']} and \ref{['fig:target2']} shows the target arrangement example. Due to the holes in the target, it is robust to wind in outdoor environments while remaining detectable by LiDAR.
• Figure 4: Visualization of the cropping process. \ref{['fig:sub2']} is cropped automatically from \ref{['fig:sub1']} based on the GPS target pose. Ground points and outliers are manually removed from \ref{['fig:sub2']} to obtain \ref{['fig:sub3']}. All black boxes indicate the same area that contains a single target.
• Figure 5: Example of target plane estimation in step by step. The red and blue points in \ref{['fig:kmean']} and \ref{['fig:ransac']} indicate the identified target planes respectively.