Perception Characteristics Distance: Measuring Stability and Robustness of Perception System in Dynamic Conditions under a Certain Decision Rule

TL;DR

Perception Characteristics Distance (PCD) provides an uncertainty-aware measure of perception performance, supporting safer and more robust ADS operation, while the SensorRainFall dataset offers a valuable benchmark for evaluation.

Abstract

The safety of autonomous driving systems (ADS) depends on accurate perception across distance and driving conditions. The outputs of AI perception algorithms are stochastic, which have a major impact on decision making and safety outcomes, including time-to-collision estimation. However, current perception evaluation metrics do not reflect the stochastic nature of perception algorithms. We introduce the Perception Characteristics Distance (PCD), a novel metric incorporating model output uncertainty as represented by the farthest distance at which an object can be reliably detected. To represent a system's overall perception capability in terms of reliable detection distance, we average PCD values across multiple detection quality and probabilistic thresholds to produce the average PCD (aPCD). For empirical validation, we present the SensorRainFall dataset, collected on the Virginia Smart Road using a sensor-equipped vehicle (cameras, radar, and LiDAR) under different weather (clear and rainy) and illumination conditions (daylight, streetlight, and nighttime). The dataset includes ground-truth distances, bounding boxes, and segmentation masks for target objects. Experiments with state-of-the-art models show that aPCD captures meaningful differences across weather, daylight, and illumination conditions, which traditional evaluation metrics fail to reflect. PCD provides an uncertainty-aware measure of perception performance, supporting safer and more robust ADS operation, while the SensorRainFall dataset offers a valuable benchmark for evaluation. The SensorRainFall dataset is publicly available at https://www.kaggle.com/datasets/datadrivenwheels/sensorrainfall, and the evaluation code is available at https://github.com/datadrivenwheels/PCD_Python.

Figures (10)

• Figure 1: Comparison of detection confidence at near vs. far ranges. Near-range object detections with consistently high confidence scores ($\geq$ 0.90). Far-range detections exhibit unstable and fluctuating confidence, including brief drops as low as 0.24. Despite visual continuity of the object, detection reliability degrades with increasing distance.
• Figure 2: SensorRainfall dataset example.
• Figure 3: Conceptual illustration of PCD.
• Figure 4: PCD of Grounding DINO for vehicle detection under clear and rainy daylight (threshold: $p^{thres}$ = 0.5 and $y^{thres}$ = 0.5).
• Figure 5: Evaluation results of Example 2: Object detection, Vehicle, Rainy night. GLIP vs. DyHead (PCD threshold: $p^{thres}$ = 0.3 and $y^{thres}$ = 0.3).