Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating Roadside Perception for Autonomous Vehicles: Insights from Field Testing

Rusheng Zhang, Depu Meng, Shengyin Shen, Tinghan Wang, Tai Karir, Michael Maile, Henry X. Liu

TL;DR

This work addresses the lack of standardized evaluation methods for roadside perception systems by proposing a field-tested evaluation framework that combines measurement techniques, a multi-metric evaluation suite, and a structured trial design. The methodology is demonstrated in Mcity with three off-the-shelf roadside perception systems, including a LiDAR-based and two image-based solutions, to quantify latency, positioning error, and tracking performance using MOTP, MOTA, IDF1, and HOTA metrics. Results show a clear advantage for LiDAR-based systems in localization and overall tracking, with image-based systems suffering from higher latency and localization errors under strict 1.5 m thresholds, especially in day/night conditions. The findings support the development of standardized benchmarks for roadside perception and highlight areas for improvement in infrastructure-based sensing, latency stability, and localization accuracy to enable reliable cooperative driving solutions.

Abstract

Roadside perception systems are increasingly crucial in enhancing traffic safety and facilitating cooperative driving for autonomous vehicles. Despite rapid technological advancements, a major challenge persists for this newly arising field: the absence of standardized evaluation methods and benchmarks for these systems. This limitation hampers the ability to effectively assess and compare the performance of different systems, thus constraining progress in this vital field. This paper introduces a comprehensive evaluation methodology specifically designed to assess the performance of roadside perception systems. Our methodology encompasses measurement techniques, metric selection, and experimental trial design, all grounded in real-world field testing to ensure the practical applicability of our approach. We applied our methodology in Mcity\footnote{\url{https://mcity.umich.edu/}}, a controlled testing environment, to evaluate various off-the-shelf perception systems. This approach allowed for an in-depth comparative analysis of their performance in realistic scenarios, offering key insights into their respective strengths and limitations. The findings of this study are poised to inform the development of industry-standard benchmarks and evaluation methods, thereby enhancing the effectiveness of roadside perception system development and deployment for autonomous vehicles. We anticipate that this paper will stimulate essential discourse on standardizing evaluation methods for roadside perception systems, thus pushing the frontiers of this technology. Furthermore, our results offer both academia and industry a comprehensive understanding of the capabilities of contemporary infrastructure-based perception systems.

Evaluating Roadside Perception for Autonomous Vehicles: Insights from Field Testing

TL;DR

This work addresses the lack of standardized evaluation methods for roadside perception systems by proposing a field-tested evaluation framework that combines measurement techniques, a multi-metric evaluation suite, and a structured trial design. The methodology is demonstrated in Mcity with three off-the-shelf roadside perception systems, including a LiDAR-based and two image-based solutions, to quantify latency, positioning error, and tracking performance using MOTP, MOTA, IDF1, and HOTA metrics. Results show a clear advantage for LiDAR-based systems in localization and overall tracking, with image-based systems suffering from higher latency and localization errors under strict 1.5 m thresholds, especially in day/night conditions. The findings support the development of standardized benchmarks for roadside perception and highlight areas for improvement in infrastructure-based sensing, latency stability, and localization accuracy to enable reliable cooperative driving solutions.

Abstract

Roadside perception systems are increasingly crucial in enhancing traffic safety and facilitating cooperative driving for autonomous vehicles. Despite rapid technological advancements, a major challenge persists for this newly arising field: the absence of standardized evaluation methods and benchmarks for these systems. This limitation hampers the ability to effectively assess and compare the performance of different systems, thus constraining progress in this vital field. This paper introduces a comprehensive evaluation methodology specifically designed to assess the performance of roadside perception systems. Our methodology encompasses measurement techniques, metric selection, and experimental trial design, all grounded in real-world field testing to ensure the practical applicability of our approach. We applied our methodology in Mcity\footnote{\url{https://mcity.umich.edu/}}, a controlled testing environment, to evaluate various off-the-shelf perception systems. This approach allowed for an in-depth comparative analysis of their performance in realistic scenarios, offering key insights into their respective strengths and limitations. The findings of this study are poised to inform the development of industry-standard benchmarks and evaluation methods, thereby enhancing the effectiveness of roadside perception system development and deployment for autonomous vehicles. We anticipate that this paper will stimulate essential discourse on standardizing evaluation methods for roadside perception systems, thus pushing the frontiers of this technology. Furthermore, our results offer both academia and industry a comprehensive understanding of the capabilities of contemporary infrastructure-based perception systems.
Paper Structure (31 sections, 1 theorem, 25 equations, 6 figures, 8 tables)

This paper contains 31 sections, 1 theorem, 25 equations, 6 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Problem Formulation
  4. Methodology
  5. Measurement Techniques
  6. Assumptions and Mathematical Modeling
  7. Latency Measurement
  8. Estimating Positioning Error
  9. Metrics Selection
  10. Data Representation and Preprocessing
  11. Matching Mechanisms
  12. Point Matching
  13. Association Matching:
  14. True Positives, False Positives, False Negatives, and ID Switches
  15. Multiple Object Tracking Precision (MOTP)
  16. ...and 16 more sections

Key Result

Theorem 1

The estimator $\bm{\Tilde{e_d}}$ is an unbiased estimator of the positional error.

Figures (6)

  • Figure 1: Illustrations of the experimental setup for roadside perception systems testing in Mcity.
  • Figure 2: Depiction of the experimental setup for latency measurement, showcasing the vehicle's movement path involving acceleration, constant speed, and deceleration areas.
  • Figure 3: Illustration of True Positives, False Positives, False Negatives, and Identity Switches in the context of our matching process.
  • Figure 4: Illustrations of all trials designed for evaluation. (a) Latency trials, (b) One Vehicle Trials, (c) One Vehicle with Pedestrian Trials, and (d), (e), (f) Two Vehicles with Pedestrian Trials.
  • Figure 5: The detection results of the System A (first row), the System B (second row), and the System C (third row) in different trials. The red dots are detection results and the blue dots are ground truth.
  • ...and 1 more figures

Theorems & Definitions (2)

  • Theorem 1
  • proof