BEACON: A Bayesian Evolutionary Approach for Counterexample Generation of Control Systems

TL;DR

BEACON is presented, a novel framework that enhances the falsification process through a combination of Bayesian optimization and covariance matrix adaptation evolutionary strategy, which offers a promising direction for achieving thorough and resource-efficient safety evaluations, ensuring the reliability of control systems in critical applications.

Abstract

The rigorous safety verification of control systems in critical applications is essential, given their increasing complexity and integration into everyday life. Simulation-based falsification approaches play a pivotal role in the safety verification of control systems, particularly within critical applications. These methods systematically explore the operational space of systems to identify configurations that result in violations of safety specifications. However, the effectiveness of traditional simulation-based falsification is frequently limited by the high dimensionality of the search space and the substantial computational resources required for exhaustive exploration. This paper presents BEACON, a novel framework that enhances the falsification process through a combination of Bayesian optimization and covariance matrix adaptation evolutionary strategy. By exploiting quantitative metrics to evaluate how closely a system adheres to safety specifications, BEACON advances the state-of-the-art in testing methodologies. It employs a model-based test point selection approach, designed to facilitate exploration across dynamically evolving search zones to efficiently uncover safety violations. Our findings demonstrate that BEACON not only locates a higher percentage of counterexamples compared to standalone BO but also achieves this with significantly fewer simulations than required by CMA-ES, highlighting its potential to optimize the verification process of control systems. This framework offers a promising direction for achieving thorough and resource-efficient safety evaluations, ensuring the reliability of control systems in critical applications.

Figures (4)

• Figure 1: Schematic Representation of the BEACON Falsification Framework. This framework constructs a model for evaluating system specifications within a defined local search zone, $\mathcal{U}_L \subseteq$$\mathcal{U}_G$, as highlighted by the red box. Over $P$ iterations, BO is used to select new environmental parameters for simulation within $\mathcal{U}_L$. Upon exhausting the iteration budget $P$, the framework uses the $P_{\text{best }}$ environmental parameters to derive the mean and standard deviation, using principles from the CMA-ES, as indicated in blue. These statistical measures are then used to determine the upper and lower bounds of the subsequent local search zone, setting the stage for the next cycle of the process.
• Figure 2: Illustration of the BEACON methodology applied within a $2$-dimensional global search space $\mathcal{U}_{G}=[0,20]^2$. Each subfigure shows the evolving boundaries of local search zones, with the highlighted points representing the $P_{best}$ environmental parameters selected to refine the subsequent search space. This sequential adaptation showcases the framework's progression through the search space to efficiently explore regions of interest.
• Figure 3: The illustration of violation rate vs. simulation budget for the mountain car, automatic transmission, neural network, F16, and air fuel control case studies. In these plots, BEACON's results are presented in blue, and BO's results are presented in red. BEACON performs better than BO at lower simulation budgets in the cases of mountain car and automatic transmission. In the air fuel control case study, BEACON and BO performed similarly across each budget. In the neural network and F-$16$ environments, BEACON achieves higher violation rates for each budget than BO.
• Figure 4: Comparative analysis of violation rates across case studies.