Finding Unknown Unknowns using Cyber-Physical System Simulators (Extended Report)

TL;DR

The paper tackles finding unknown unknowns in cyber-physical systems by analyzing mode sequences produced by a black-box CPS simulator. It introduces the Convex Mode Sequence Assumption and models input points as convex regions corresponding to mode sequences, enabling skip-based sampling. Two accelerated testing algorithms, Convex Rejection Sampling (CRS) and Region Distance Maximization (RDM), select new inputs outside existing regions to maximize the number of distinct mode sequences $|Y_{\\kappa}|$ under a simulation budget $\\kappa$. Empirical results across Voronoi, Navigation, Gearbox Meshing, and Automatic Transmission benchmarks show substantial speedups over random sampling (up to tens to hundreds of times in some cases) and greater discovery of rare behaviors. The approach is specification-free and complements existing verification, falsification, and coverage-based testing methods in CPS analysis.

Abstract

Simulation-based approaches are among the most practical means to search for safety violations, bugs, and other unexpected events in cyber-physical systems (CPS). Where existing approaches search for simulations violating a formal specification or maximizing a notion of coverage, in this work we propose a new goal for testing: to discover unknown rare behaviors by examining discrete mode sequences. We assume a CPS simulator outputs mode information, and strive to explore the sequences of modes produced by varying the initial state or time-varying uncertainties. We hypothesize that rare mode sequences are often the most interesting to a designer, and we develop two accelerated sampling algorithms that speed up the process of finding such sequences. We evaluate our approach on several benchmarks, ranging from synthetic examples to Simulink diagrams of a CPS, demonstrating in some cases a speedup of over 100x compared with a random sampling strategy.

Figures (8)

• Figure 1: A 2-dimensional Voronoi diagram created using the given Gaussian distribution, representing the input space for the Voronoi system. A few large $\text{region}$s take up the majority of the input space, while many smaller $\text{region}$s occupy a small portion in the top-right.
• Figure 2: Results for the Voronoi system.
• Figure 3: A rare behavior in one of the navigation benchmarks, where the object transitions out of the initial cell to the right, then proceeds clockwise to the terminal cell.
• Figure 4: Mode sequence discovery rates for the NAV 10 benchmark. $\text{RDM}$ was able to discover more than twice as many distinct behaviors in 20x fewer simulations than random sampling.
• Figure 5: An example trajectory of the gearbox meshing benchmark. The sleeve bounces once against the upper tooth, then twice against the lower tooth, before successfully meshing with the gear; thus, the mode sequence of this simulation is $\texttt{2,1,1,3}$.