Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Comprehensive Assessment of LiDAR Evaluation Metrics: A Comparative Study Using Simulated and Real Data

Syed Mostaquim Ali, Taufiq Rahman, Ghazal Farhani, Mohamed H. Zaki, Benoit Anctil, Dominique Charlebois

TL;DR

This work tackles the challenge of validating virtual testing environments for autonomous driving by developing and evaluating geometric LiDAR similarity metrics. It identifies Density Aware Chamfer Distance (DCD) as the most robust metric across noise, density, distortion, and sensor settings, and demonstrates a practical two-step approach (DCD with $\alpha=1$ followed by Chamfer Distance) to assess LiDAR realism. The authors build a CARLA-based digital twin from real-world data, generate simulated LiDAR scans under varying ODDs, and show that intensity correction significantly improves perception-model alignment, achieving a mean IoU increase from 4.7% to 21.2%. The study highlights the importance of geometry-driven benchmarks for VTE credibility while acknowledging limitations in reflectivity/intensity modeling and vegetation representation, and points to future directions in richer mesh generation and multimodal validation.

Abstract

For developing safe Autonomous Driving Systems (ADS), rigorous testing is required before they are deemed safe for road deployments. Since comprehensive conventional physical testing is impractical due to cost and safety concerns, Virtual Testing Environments (VTE) can be adopted as an alternative. Comparing VTE-generated sensor outputs against their real-world analogues can be a strong indication that the VTE accurately represents reality. Correspondingly, this work explores a comprehensive experimental approach to finding evaluation metrics suitable for comparing real-world and simulated LiDAR scans. The metrics were tested in terms of sensitivity and accuracy with different noise, density, distortion, sensor orientation, and channel settings. From comparing the metrics, we found that Density Aware Chamfer Distance (DCD) works best across all cases. In the second step of the research, a Virtual Testing Environment was generated using real LiDAR scan data. The data was collected in a controlled environment with only static objects using an instrumented vehicle equipped with LiDAR, IMU and cameras. Simulated LiDAR scans were generated from the VTEs using the same pose as real LiDAR scans. The simulated and LiDAR scans were compared in terms of model perception and geometric similarity. Actual and simulated LiDAR scans have a similar semantic segmentation output with a mIoU of 21\% with corrected intensity and an average density aware chamfer distance (DCD) of 0.63. This indicates a slight difference in the geometric properties of simulated and real LiDAR scans and a significant difference between model outputs. During the comparison, density-aware chamfer distance was found to be the most correlated among the metrics with perception methods.

Comprehensive Assessment of LiDAR Evaluation Metrics: A Comparative Study Using Simulated and Real Data

TL;DR

This work tackles the challenge of validating virtual testing environments for autonomous driving by developing and evaluating geometric LiDAR similarity metrics. It identifies Density Aware Chamfer Distance (DCD) as the most robust metric across noise, density, distortion, and sensor settings, and demonstrates a practical two-step approach (DCD with followed by Chamfer Distance) to assess LiDAR realism. The authors build a CARLA-based digital twin from real-world data, generate simulated LiDAR scans under varying ODDs, and show that intensity correction significantly improves perception-model alignment, achieving a mean IoU increase from 4.7% to 21.2%. The study highlights the importance of geometry-driven benchmarks for VTE credibility while acknowledging limitations in reflectivity/intensity modeling and vegetation representation, and points to future directions in richer mesh generation and multimodal validation.

Abstract

For developing safe Autonomous Driving Systems (ADS), rigorous testing is required before they are deemed safe for road deployments. Since comprehensive conventional physical testing is impractical due to cost and safety concerns, Virtual Testing Environments (VTE) can be adopted as an alternative. Comparing VTE-generated sensor outputs against their real-world analogues can be a strong indication that the VTE accurately represents reality. Correspondingly, this work explores a comprehensive experimental approach to finding evaluation metrics suitable for comparing real-world and simulated LiDAR scans. The metrics were tested in terms of sensitivity and accuracy with different noise, density, distortion, sensor orientation, and channel settings. From comparing the metrics, we found that Density Aware Chamfer Distance (DCD) works best across all cases. In the second step of the research, a Virtual Testing Environment was generated using real LiDAR scan data. The data was collected in a controlled environment with only static objects using an instrumented vehicle equipped with LiDAR, IMU and cameras. Simulated LiDAR scans were generated from the VTEs using the same pose as real LiDAR scans. The simulated and LiDAR scans were compared in terms of model perception and geometric similarity. Actual and simulated LiDAR scans have a similar semantic segmentation output with a mIoU of 21\% with corrected intensity and an average density aware chamfer distance (DCD) of 0.63. This indicates a slight difference in the geometric properties of simulated and real LiDAR scans and a significant difference between model outputs. During the comparison, density-aware chamfer distance was found to be the most correlated among the metrics with perception methods.

Paper Structure

This paper contains 50 sections, 4 equations, 23 figures, 1 table.

Table of Contents

  1. Introduction
  2. Literature Review
  3. LiDAR Sensing Technology
  4. Software
  5. Credibility Assessment Framework
  6. LiDAR simulations
  7. Research Gap
  8. Proposed Qualification Protocol
  9. Generation of Ground-Truth Driving Environment
  10. Generation of Simulated Sensor Data Under Different ODDs
  11. Development of Comparison Metrics
  12. Developing Evaluation Metrics
  13. Conditions for Evaluation metrics
  14. Generating Cases to Evaluate Similarity Metrics
  15. Channel settings
  16. ...and 35 more sections

Figures (23)

  • Figure 1: Simulated LiDAR scans of different channel settings from left VLP-16, VLP-32c, HDL-64E (top plots provide the spatial channel distribution for different LiDAR models)
  • Figure 2: Simulated LiDAR scans with different orientations
  • Figure 3: Simulated LiDAR scans with different distortions
  • Figure 4: Added Noise to Real LiDAR Scans. In this example the magnitude of the noise is 0.2 normal standard deviation.
  • Figure 5: Added Random Scattered points into real LiDAR scans. The figure shows 10,000 random outlier points within the point cloud bounds.
  • ...and 18 more figures