Automated Test Validators for Flaky Cyber-Physical System Simulators: Approach and Evaluation

TL;DR

The paper tackles the inefficiency and unreliability of simulation-based CPS testing by introducing assertion-based test validators that filter out inputs likely not to meaningfully exercise the SUT. It presents GenTV, a data-driven pipeline that can generate validators via genetic programming with SBFL-based fitness (notably Ochiai) or via interpretable ML (DT/DR), plus a pruning step to ensure verdict consistency. The approach is extended to signal-based CPS with a formal logic L that can express common CPS properties and is shown to align with a large portion of industrial requirements. Empirical results across diverse case studies demonstrate high accuracy and robustness to simulator flakiness, with strong alignment to preconditions and ODD limits, suggesting substantial execution-time savings in practice.

Abstract

Simulation-based testing of cyber-physical systems (CPS) is costly due to the time-consuming execution of CPS simulators. In addition, CPS simulators may be flaky, leading to inconsistent test outcomes and requiring repeated test re-execution for reliable test verdicts. Many test inputs within the input space of CPS may not effectively exercise the behaviour of the system under test (SUT) -- for instance, those that violate system preconditions, exceed operational design domain (ODD) limits, or represent inherently safe scenarios. In this article, we propose to use test validators to filter out such test inputs before execution. We describe two methods for generating test validators: one using genetic programming (GP) that employs well-known spectrum-based fault localization (SBFL) ranking formulas, namely Ochiai, Tarantula, and Naish, as fitness functions; and the other using decision trees (DT) and decision rules (DR). We evaluate our test validators through case studies in the domains of aerospace, networking and autonomous driving. We show that test validators generated using GP with Ochiai are significantly more accurate than those generated using GP with Tarantula and Naish or using DT or DR. Moreover, this accuracy advantage remains even when accounting for the flakiness of the simulator. We further show that our test validators generated by GP with Ochiai are robust against flakiness with only 4% average variation in their accuracy results across four different network and autonomous-driving systems with flaky behaviours. Finally, we show that, on average, 88.7% of the assertions inferred by our approach align or overlap with requirements precondition violations, ODD-limit violations, and nominal safe conditions extracted from technical standards and empirical results in the literature.

Figures (13)

• Figure 1: An assertion-based test validator for a simplified ADS with inputs $\mathit{speed}$, $\mathit{time\_of\_day}$ and $\mathit{traffic\_density}$, along with an example set of test inputs for this system. In this figure, ✓ indicates the pass verdict, ✗ indicates the fail verdict and ? indicates that the test validator is inconclusive.
• Figure 2: Our approach for deriving test validators, GenTV.
• Figure 3: Syntactic rules of the grammar (denoted by $\mathcal{G}$) that define the assertions over system input variables. The symbol $|$ separates alternatives, const is an ephemeral random constant generator and cp represents an input variable of the SUT.
• Figure 4: Fitness functions for GenTV, adopted from the spectrum-based fault localization (SBFL) literature.The $c_p (c)$ and $c_f (c)$ functions respectively represent the number of passing and failing tests in the training set $\mathit{TS}$ satisfying the condition $c$. The $\mathit{TS}_f$ and $\mathit{TS}_p$ sets indicate the set of failing and passing tests in $\mathit{TS}$, respectively. That is, $\mathit{TS} = \mathit{TS}_p \cup \mathit{TS}_f$.
• Figure 5: Computing $c_p(c)$ and $c_f(c)$ in SBFL fitness functions in Figure \ref{['fig:rankingmetrics']} for our GP-based condition inference

Theorems & Definitions (4)

• Definition 1: Test validator
• Definition 2: Verdict threshold
• Definition 3: Bipartite Graph based on Assertion Conditions
• proof