Boundary State Generation for Testing and Improvement of Autonomous Driving Systems

TL;DR

This work tackles the problem of dependable autonomous driving system testing under resource constraints by focusing on boundary conditions within a fixed environment. It introduces GENBO, a boundary-state generator that mutates the ego-vehicle state to locate boundary state pairs and uses a binary-search-based strategy to cross the decision boundary efficiently. The authors demonstrate that boundary-state pairs exist even for well-trained systems, that these pairs discriminate model quality, and that retraining with expert-labeled boundary data yields substantial improvements in evaluation performance (up to 3x on average). By avoiding environment changes and instead mutating driving conditions, GENBO offers a data-efficient approach to expose hidden failure modes and enhance generalization through targeted retraining.

Abstract

Recent advances in Deep Neural Networks (DNNs) and sensor technologies are enabling autonomous driving systems (ADSs) with an ever-increasing level of autonomy. However, assessing their dependability remains a critical concern. State-of-the-art ADS testing approaches modify the controllable attributes of a simulated driving environment until the ADS misbehaves. In such approaches, environment instances in which the ADS is successful are discarded, despite the possibility that they could contain hidden driving conditions in which the ADS may misbehave. In this paper, we present GENBO (GENerator of BOundary state pairs), a novel test generator for ADS testing. GENBO mutates the driving conditions of the ego vehicle (position, velocity and orientation), collected in a failure-free environment instance, and efficiently generates challenging driving conditions at the behavior boundary (i.e., where the model starts to misbehave) in the same environment instance. We use such boundary conditions to augment the initial training dataset and retrain the DNN model under test. Our evaluation results show that the retrained model has, on average, up to 3x higher success rate on a separate set of evaluation tracks with respect to the original DNN model.

Figures (5)

• Figure 1: Overview of our approach. Our approach takes an Autonomous Driving System (ADS) as input and features two steps, namely ❶ Boundary State Pairs Search and ❷ Driving Model Improvement. The first step ❶ looks for boundary state pairs of the driving model controlling the ADS, while the second step ❷ retrains the driving model using data labeled by an expert pilot driving on boundary state pairs. The output of the approach is an ADS improved by the boundary state pairs.
• Figure 2: Visualization of a vehicle state. A vehicle state is defined by three key properties, namely the position of the vehicle inside the driving track $\mathbf{p}$, the orientation of the vehicle $\psi$, and the magnitude of the velocity vector $v = | \mathbf{v} |$.
• Figure 3: Example of overlap between closeness and validity constraints concerning the "mutate orientation" function. Assuming that the orientation angle $\psi$ increases with the $x$ direction, the two ranges overlap, determining the new range for the mutated orientation $\hat{\psi}_i$ (delimited by the two dashed lines).
• Figure 4: Example of binary search to find a boundary state pair. The initial seed state is the pair $(s_{1_0}, s_{2_0})$, with $s_{2_0}$ being the more challenging state of the pair. The initial pair is consecutively mutated $L = 4$ times (difficulty increases going towards the right). Binary search efficiently finds the pair that sits on the boundary, i.e., $(s_{1_2}, s_{2_2})$
• Figure 5: Examples of boundary state pairs for the model $M_{\textit{last}}$. The two blue lines delimit the track $T$, the green plus sign is the state of the pair $s_1$ where the driving model succeeds, the red cross indicates the state $s_2$ where the driving model fails, and black dots are the closest waypoints (in terms of position within $T$) to the pair. The green and red arrows are the velocity vectors (their length is normalized and scaled 4$\times$ for visualization purposes).

Theorems & Definitions (4)

• Definition 1: State
• Definition 2: State Validity
• Definition 3: Boundary State Pair
• Definition 4: State Mutation Function