Open-Source Tool Based Framework for Automated Performance Evaluation of an AD Function

TL;DR

The paper tackles how road-network geometry influences automated driving function performance by constructing an open-source, scenario-based virtual testing pipeline. It combines OpenDRIVE/Lanelet2/OpenSCENARIO standards with tools such as Road Generation Tool, CommonRoad, ROS, and CARLA to automate scenario generation, simulation, and KPI-based evaluation. Through curved-road, T-junction, and complex-road templates, it demonstrates that geometry materially affects safety and comfort metrics, with notable differences across turning directions and road complexity. The framework offers a reproducible, extensible platform for safety validation of ADFs and informs road topology design and testing practices.

Abstract

As automation in the field of automated driving (AD) progresses, ensuring the safety and functionality of AD functions (ADFs) becomes crucial. Virtual scenario-based testing has emerged as a prevalent method for evaluating these systems, allowing for a wider range of testing environments and reproducibility of results. This approach involves AD-equipped test vehicles operating within predefined scenarios to achieve specific driving objectives. To comprehensively assess the impact of road network properties on the performance of an ADF, varying parameters such as intersection angle, curvature and lane width is essential. However, covering all potential scenarios is impractical, necessitating the identification of feasible parameter ranges and automated generation of corresponding road networks for simulation. Automating the workflow of road network generation, parameter variation, simulation, and evaluation leads to a comprehensive understanding of an ADF's behavior in diverse road network conditions. This paper aims to investigate the influence of road network parameters on the performance of a prototypical ADF through virtual scenario-based testing, ultimately advocating the importance of road topology in assuring safety and reliability of ADFs.

Figures (6)

• Figure 1: Overview of the implemented automated scenario-based testing methodology. All components and formats are open-source.
• Figure 2: The 6 layer model to structure a scenario according to SCH21.
• Figure 3: Concrete examples of the three utilized road network templates. Exemplary routes are drawn in each scenario.
• Figure 4: Illustration of the overall performance evaluation for all executed simulations clustered by the three main templates. The spider plot indicates that the ADF generally have good jerk values and almost always stays within the lane boundaries. Reaching the target seems to be an issue which might be controller related since the position is overshot. In the complex road network, acceleration and steady lane keeping values tend to be bad which may be due to the transitions between different road elements.
• Figure 5: Filtered average of the dynamic KPIs in left turns with three fixed lane width over different curve radii. For each lane width, a critical radius was identified below which the ADF failed to succeed (dashed vertical lines). Expectable, the performance in wider lanes seems to be better.