Testing CPS with Design Assumptions-Based Metamorphic Relations and Genetic Programming

TL;DR

The paper tackles the CPS testing problem when requirements apply to simple input shapes, proposing a design-assumptions-based metamorphic testing framework that leverages linear system theory. It defines metamorphic relations from the linear time-invariant design assumptions, and uses a distance-based MR-falsification degree to act as an oracle. A genetic programming approach evolves MR compositions to generate CPS input traces of arbitrary shape while constraining the control error, demonstrated on three CPS subjects. Empirical results show the GP-based MR-falsification outperforms random generation and provides informative signals beyond control error, supporting its use for discovering subtle, nontrivial failures in CPS software-hardware interactions. The method offers a domain-agnostic path to test generation and richer oracle signals, with potential extensions to environmental inputs and additional design assumptions.

Abstract

Cyber-Physical Systems (CPSs) software is used to enforce desired behaviours on physical systems. To test the interaction between the CPS software and the system's physics, engineers provide traces of desired physical states and observe traces of the actual physical states. CPS requirements describe how closely the actual physical traces should track the desired traces. These requirements are typically defined for specific, simple input traces such as step or ramp sequences, and thus are not applicable to arbitrary inputs. This limits the availability of oracles for CPSs. Our recent work proposes an approach to testing CPS using control-theoretical design assumptions instead of requirements. This approach circumvents the oracle problem by leveraging the control-theoretical guarantees that are provided when the design assumptions are satisfied. To address the test case generation and oracle problems, researchers have proposed metamorphic testing, which is based on the study of relations across tests, i.e., metamorphic relations (MRs). In this work, we define MRs based on the design assumptions and explore combinations of these MRs using genetic programming to generate CPS test cases. This enables the generation of CPS input traces with potentially arbitrary shapes, together with associated expected output traces. We use the deviation from the expected output traces to guide the generation of input traces that falsify the MRs. Our experiment results show that the MR-falsification provides engineers with new information, helping them identify passed and failed test cases. Furthermore, we show that the generation of traces that falsify the MRs is a non-trivial problem, which is successfully addressed by our genetic search.

Figures (7)

• Figure 1: Structure of a CPS. The cyber part is represented by the dashed azure box; the physical part is represented by the solid purple box; arrows denote input or output signals.
• Figure 2: Examples of step responses of the altitude control of a drone, with the reference input altitude drawn using a solid black and the actual altitude drawn with a dashed purple line. The left-hand side plots show a step response, where the steady-state error and overshoot are well defined. The right-hand side plots show instead an arbitrary input trace, for which the steady state and overshoot cannot be assessed.
• Figure 3: Example of MRs applications for a simplified drone altitude control example. Starting from the two initial tests in the first row ($r_x$ and $r_y$) we build three follow-up tests in the bottom row, showcasing MR1, MR2, and MR3, respectively. For the initial tests, we have only the input (black solid line) and output (purple dashed line) traces. Thanks to the MRs, for the follow up test cases we can also compute the expected output (dash-dotted lines) traces.
• Figure 4: Use of the metamorphic relations in the proposed approach. Boxes represent the main steps of our approach; solid arrows represent the tests' input and output traces. Specifically, black arrows are the input traces, while the purple and azure arrows are the actual and expected output traces, respectively. The dashed arrows represent the distance metrics used to quantify the control error and the MR-falsification degree.
• Figure 5: The four different patterns used for the definition of the initial traces. Each pattern has a number of time parameters (the $t_i$ indices) that are randomly selected when the ephemeral constant is created for a initial trace terminal symbol. In contrast, $b_{\mathit{amp}}$ is fixed and set by the user to ensure that the initial trace belongs to the design scope.