Make Full Use of Testing Information: An Integrated Accelerated Testing and Evaluation Method for Autonomous Driving Systems

TL;DR

This work tackles ADS safety verification under SOTIF by addressing the dimensionality of logical scenario spaces with an Integrated Accelerated Testing and Evaluation Method (ITEM). ITEM unifies testing and evaluation through an MCTS-based framework and dual surrogates, enhanced by a boundary-focused UCB and a convergence-based stopping condition. Hazardous domains are identified using a tree-structured partition of the space and hyper-cuboid approximations, quantified by API and ADI metrics. Experiments on synthetic multimodal functions and a 3D cut-in scenario show ITEM achieves higher domain identification accuracy and robust performance across dimensions, while maintaining or improving testing efficiency. The method holds promise for more accurate and scalable safety evaluation of autonomous driving systems and automates stopping to conserve computation.

Abstract

Testing and evaluation is an important step before the large-scale application of the autonomous driving systems (ADSs). Based on the three level of scenario abstraction theory, a testing can be performed within a logical scenario, followed by an evaluation stage which is inputted with the testing results of each concrete scenario generated from the logical parameter space. During the above process, abundant testing information is produced which is beneficial for comprehensive and accurate evaluations. To make full use of testing information, this paper proposes an Integrated accelerated Testing and Evaluation Method (ITEM). Based on a Monte Carlo Tree Search (MCTS) paradigm and a dual surrogates testing framework proposed in our previous work, this paper applies the intermediate information (i.e., the tree structure, including the affiliation of each historical sampled point with the subspaces and the parent-child relationship between subspaces) generated during the testing stage into the evaluation stage to achieve accurate hazardous domain identification. Moreover, to better serve this purpose, the UCB calculation method is improved to allow the search algorithm to focus more on the hazardous domain boundaries. Further, a stopping condition is constructed based on the convergence of the search algorithm. Ablation and comparative experiments are then conducted to verify the effectiveness of the improvements and the superiority of the proposed method. The experimental results show that ITEM could well identify the hazardous domains in both low- and high-dimensional cases, regardless of the shape of the hazardous domains, indicating its generality and potential for the safety evaluation of ADSs.

Figures (11)

• Figure 1: Information at different layers during the testing and evaluation process.
• Figure 2: Framework of the Proposed Integrated Accelerated Testing and Evaluation Method (ITEM).
• Figure 3: The stopping condition construction process.
• Figure 4: Hazardous domain identification method.
• Figure 5: Illustration of 2-dimensional multi-modal Gaussian function.