Discovering Decision Manifolds to Assure Trusted Autonomous Systems

TL;DR

This work tackles the challenge of validating complex autonomous systems where exhaustive state enumeration is infeasible by introducing an optimization-based data-generation framework that traces a decision manifold between correct and incorrect behaviors. It presents HybridPairCE, an extension of the Cross-Entropy method that produces explainable vanilla/perturbed path pairs and integrates dataset variance into the cost function to train machine-learned correctness properties (MLCP) from simulator data. Applied to a CARLA-based software-in-the-loop autonomous vehicle scenario, the approach yields high-performing MLCP models (notably Random Forests) capable of predicting potential collisions with advance notice, enabling offline risk assessment and real-time monitoring for trusted autonomy. Key contributions include the CE-based data-generation pipeline with variance-aware weighting, the concept and evaluation of MLCPs, and empirical evidence that explainable MLCP can support continuous assurance and safer deployment of autonomous CPS. The results demonstrate practical potential for integrating MLCP into verification, validation, and monitoring workflows to enhance trust in autonomous systems throughout their lifecycle.

Abstract

Developing and fielding complex systems requires proof that they are reliably correct with respect to their design and operating requirements. Especially for autonomous systems which exhibit unanticipated emergent behavior, fully enumerating the range of possible correct and incorrect behaviors is intractable. Therefore, we propose an optimization-based search technique for generating high-quality, high-variance, and non-trivial data which captures the range of correct and incorrect responses a system could exhibit. This manifold between desired and undesired behavior provides a more detailed understanding of system reliability than traditional testing or Monte Carlo simulations. After discovering data points along the manifold, we apply machine learning techniques to quantify the decision manifold's underlying mathematical function. Such models serve as correctness properties which can be utilized to enable both verification during development and testing, as well as continuous assurance during operation, even amidst system adaptations and dynamic operating environments. This method can be applied in combination with a simulator in order to provide evidence of dependability to system designers and users, with the ultimate aim of establishing trust in the deployment of complex systems. In this proof-of-concept, we apply our method to a software-in-the-loop evaluation of an autonomous vehicle.

Figures (9)

• Figure 1: In other work, we applied our approach to the Defense Threat Reduction Agency's (DTRA) Mission Impact of Nuclear Effects Software (MINES) to discover a decision manifold for military nuclear survivability. This technique is applicable to any complex system, including autonomous systems, with a realistic simulation capability.
• Figure 2: The relationship between correctness properties and its various sub-types.
• Figure 3: A visual depiction of the execution traces used to train the model. Note the proximity of the vanilla/perturbed hybrid-pairs to the decision boundary, and the larger relative distance of the variant and rudimentary traces.
• Figure 4: A visual depiction of a vanilla/perturbed paths with the ego and adversary trajectories. Note the similarity in position of the vanilla/perturbed paths of the adversary prior to the collision. The position of each vehicle is represented by grid coordinate, where each grid cell is approximately the length of a vehicle and the width of one half of a traffic lane.
• Figure 5: The same scenario represented in Fig. \ref{['fig:path_vis_trunc_labeled']} rendered in the simulator. The left column shows the vanilla adversary path, and the right column shows a perturbed path, where the perturbed adversary path results in a collision with the ego vehicle at $t=4s$.