Scenario-based assessment of automated driving systems: How (not) to parameterize scenarios?

TL;DR

The paper addresses how parameterizing UN R157's three ADS evaluation scenarios (cut-in, cut-out, LVD) influences simulation outcomes and argues that parameterization choice can skew assessments of automated driving systems. It introduces a method to compare non-parameterized scenarios with their parameterized counterparts, analyzes a case study using four driver models, and demonstrates that parameterization materially affects results in a way that depends on scenario type and evaluation criteria. The study finds that some parameterizations align better with non-parameterized data (improving recall, precision, and $F1$), while others can produce misleading results, underscoring the need for justification and careful design of scenario parameterization. Practically, the work suggests regulatory amendments to require justification of parameterization choices and promotes combining parameterized and non-parameterized testing to achieve robust ADS safety assessments.

Abstract

The development of Automated Driving Systems (ADSs) has advanced significantly. To enable their large-scale deployment, the United Nations Regulation 157 (UN R157) concerning the approval of Automated Lane Keeping Systems (ALKSs) has been approved in 2021. UN R157 requires an activated ALKS to avoid any collisions that are reasonably preventable and proposes a method to distinguish reasonably preventable collisions from unpreventable ones using "the simulated performance of a skilled and attentive human driver". With different driver models, benchmarks are set for ALKSs in three types of scenarios. The three types of scenarios considered in the proposed method in UN R157 assume a certain parameterization without any further consideration. This work investigates the parameterization of these scenarios, showing that the choice of parameterization significantly affects the simulation outcomes. By comparing real-world and parameterized scenarios, we show that the influence of parameterization depends on the scenario type, driver model, and evaluation criterion. Alternative parameterizations are proposed, leading to results that are closer to the non-parameterized scenarios in terms of recall, precision, and F1 score. The study highlights the importance of careful scenario parameterization and suggests improvements to the current UN R157 approach.

Figures (3)

• Figure 1: F1 scores for the cut-in parameterizations. The distance from the center represents the F1 score, starting at 0.0 in the center until a maximum of 1.0 at the outer black line. The different lines denote the different parameterizations as listed in \ref{['sec:parameterizations']}.
• Figure 2: F1 scores for the cut-out parameterizations. Parameterization \ref{['par:cut-out 2']} is hardly visible because its results are almost similar to the results of parameterization \ref{['par:cut-out 4']}.
• Figure 3: F1 scores for the LVD parameterizations.