EPSM: A Novel Metric to Evaluate the Safety of Environmental Perception in Autonomous Driving

TL;DR

Traditional perception metrics fail to capture safety implications in autonomous driving. The authors introduce EPSM, a joint offline safety metric that evaluates object and lane detection through a lightweight object safety metric (criticality, severity, weighting) and a lane safety metric, with a fusion mechanism that accounts for interdependence between tasks. EPSM yields a single final score with a five-level safety classification and is demonstrated on the DeepAccident dataset, revealing safety-critical errors that conventional metrics overlook. This approach provides an interpretable, actionable safety assessment to guide the development and evaluation of environmental perception systems for safer autonomous driving.

Abstract

Extensive evaluation of perception systems is crucial for ensuring the safety of intelligent vehicles in complex driving scenarios. Conventional performance metrics such as precision, recall and the F1-score assess the overall detection accuracy, but they do not consider the safety-relevant aspects of perception. Consequently, perception systems that achieve high scores in these metrics may still cause misdetections that could lead to severe accidents. Therefore, it is important to evaluate not only the overall performance of perception systems, but also their safety. We therefore introduce a novel safety metric for jointly evaluating the most critical perception tasks, object and lane detection. Our proposed framework integrates a new, lightweight object safety metric that quantifies the potential risk associated with object detection errors, as well as an lane safety metric including the interdependence between both tasks that can occur in safety evaluation. The resulting combined safety score provides a unified, interpretable measure of perception safety performance. Using the DeepAccident dataset, we demonstrate that our approach identifies safety critical perception errors that conventional performance metrics fail to capture. Our findings emphasize the importance of safety-centric evaluation methods for perception systems in autonomous driving.

Figures (5)

• Figure 1: Exemplary scenes including the resulting safety score $S$. Green boxes mark correct detections, red represents an object which is not detected. Orange lines represent the detected lane.
• Figure 2: Overview of the proposed EPSM and the intermediate metrics.
• Figure 3: Visualization of power mean function to combine intermediate metric scores $S_{obj}$ and $S_{lane}$ into the single intermediate score $S_p$.
• Figure 4: Visualization of the decision tree to fine-tune the intermediate safety score $S_p$
• Figure 5: Example scenario, showing the advantage of the proposed object safety metric compared to common performance metrics. Red vehicle in the center represents the ego with its detected lane (green). Other red boxes with filling represent undetected objects with their criticality.