Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Falsification of Cyber-Physical Systems using Bayesian Optimization

Zahra Ramezani, Kenan Šehić, Luigi Nardi, Knut Åkesson

TL;DR

This work tackles the expensive problem of falsifying cyber-physical systems by framing it as a Bayesian-optimization task over input signals. It introduces two practical enhancements: TuRBO, a trust-region BO approach that enables efficient high-dimensional search with local GP surrogates, and πBO, a priors-aware acquisition that injects user knowledge (e.g., corners) while maintaining convergence. Through extensive benchmarking on standard falsification problems, the authors show that TuRBO with an LCB acquisition often yields superior performance on hard instances, while πBO provides clear benefits when the dimensionality is moderate and priors are informative. The results support adopting TuRBO as an out-of-the-box tool for CPS falsification and demonstrate that incorporating prior knowledge can substantially reduce the simulation budget, with potential for practical impact in safety-critical CPS testing.

Abstract

Cyber-physical systems (CPSs) are often complex and safety-critical, making it both challenging and crucial to ensure that the system's specifications are met. Simulation-based falsification is a practical testing technique for increasing confidence in a CPS's correctness, as it only requires that the system be simulated. Reducing the number of computationally intensive simulations needed for falsification is a key concern. In this study, we investigate Bayesian optimization (BO), a sample-efficient approach that learns a surrogate model to capture the relationship between input signal parameterization and specification evaluation. We propose two enhancements to the basic BO for improving falsification: (1) leveraging local surrogate models, and (2) utilizing the user's prior knowledge. Additionally, we address the formulation of acquisition functions for falsification by proposing and evaluating various alternatives. Our benchmark evaluation demonstrates significant improvements when using local surrogate models in BO for falsifying challenging benchmark examples. Incorporating prior knowledge is found to be especially beneficial when the simulation budget is constrained. For some benchmark problems, the choice of acquisition function noticeably impacts the number of simulations required for successful falsification.

Falsification of Cyber-Physical Systems using Bayesian Optimization

TL;DR

This work tackles the expensive problem of falsifying cyber-physical systems by framing it as a Bayesian-optimization task over input signals. It introduces two practical enhancements: TuRBO, a trust-region BO approach that enables efficient high-dimensional search with local GP surrogates, and πBO, a priors-aware acquisition that injects user knowledge (e.g., corners) while maintaining convergence. Through extensive benchmarking on standard falsification problems, the authors show that TuRBO with an LCB acquisition often yields superior performance on hard instances, while πBO provides clear benefits when the dimensionality is moderate and priors are informative. The results support adopting TuRBO as an out-of-the-box tool for CPS falsification and demonstrate that incorporating prior knowledge can substantially reduce the simulation budget, with potential for practical impact in safety-critical CPS testing.

Abstract

Cyber-physical systems (CPSs) are often complex and safety-critical, making it both challenging and crucial to ensure that the system's specifications are met. Simulation-based falsification is a practical testing technique for increasing confidence in a CPS's correctness, as it only requires that the system be simulated. Reducing the number of computationally intensive simulations needed for falsification is a key concern. In this study, we investigate Bayesian optimization (BO), a sample-efficient approach that learns a surrogate model to capture the relationship between input signal parameterization and specification evaluation. We propose two enhancements to the basic BO for improving falsification: (1) leveraging local surrogate models, and (2) utilizing the user's prior knowledge. Additionally, we address the formulation of acquisition functions for falsification by proposing and evaluating various alternatives. Our benchmark evaluation demonstrates significant improvements when using local surrogate models in BO for falsifying challenging benchmark examples. Incorporating prior knowledge is found to be especially beneficial when the simulation budget is constrained. For some benchmark problems, the choice of acquisition function noticeably impacts the number of simulations required for successful falsification.
Paper Structure (32 sections, 10 equations, 3 figures, 7 tables, 1 algorithm)

This paper contains 32 sections, 10 equations, 3 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Background
  4. Falsification of Cyber-Physical Systems
  5. Signal Temporal Logic
  6. Quantitative Semantics
  7. Bayesian Optimization
  8. Method
  9. TuRBO
  10. Thompson Sampling (TS)
  11. Lower Confidence Bound (LCB)
  12. Probability of Improvement (PI)
  13. Incorporating prior belief
  14. Experimental Evaluation
  15. Experimental Setup
  16. ...and 17 more sections

Figures (3)

  • Figure 1: The process of optimization-based falsification Claessen2018.
  • Figure 2: A cactus plot showing the performance of optimization-based methods on the hard problems in Table \ref{['fig:cactus_Hard_Examples']}. The plotted values show how many successful falsifications ($x$-axis) were completed in less than always simulations ($y$-axis, logarithmic scale).
  • Figure 3: A cactus plot showing the performance of all examples. The plotted values show how many successful falsifications ($x$-axis) were completed for a given number of simulations ($y$-axis, logarithmic scale). A maximum of 1000 simulations are evaluated.