Stress Testing Control Loops in Cyber-Physical Systems

TL;DR

This paper tackles stress testing of control-based cyber-physical systems (CPSs), where control-design assumptions underlie formal guarantees that may fail in practice. It introduces a frequency-domain input-space characterization and a novel test-case parametrisation with shape, amplitude, and time-scaling, enabling systematic generation of stress tests that falsify linear-model assumptions. Metamorphic relations and two behavior metrics, the degree of non-linearity $\mathfrak{d}_{nl}$ and the degree of filtering $\mathfrak{d}_{f}$, guide test selection and interpretation, with empirical validation on drone altitude control, DC servo variants, and an aircraft model. The results demonstrate the approach’s effectiveness in exposing non-linear phenomena, revealing how actuator saturation and friction impact design scope, and providing practical insights for verification, runtime checks, and CPS redesign decisions.

Abstract

Cyber-Physical Systems (CPSs) are often safety-critical and deployed in uncertain environments. Identifying scenarios where CPSs do not comply with requirements is fundamental but difficult due to the multidisciplinary nature of CPSs. We investigate the testing of control-based CPSs, where control and software engineers develop the software collaboratively. Control engineers make design assumptions during system development to leverage control theory and obtain guarantees on CPS behaviour. In the implemented system, however, such assumptions are not always satisfied, and their falsification can lead to loss of guarantees. We define stress testing of control-based CPSs as generating tests to falsify such design assumptions. We highlight different types of assumptions, focusing on the use of linearised physics models. To generate stress tests falsifying such assumptions, we leverage control theory to qualitatively characterise the input space of a control-based CPS. We propose a novel test parametrisation for control-based CPSs and use it with the input space characterisation to develop a stress testing approach. We evaluate our approach on three case study systems, including a drone, a continuous-current motor (in five configurations), and an aircraft.Our results show the effectiveness of the proposed testing approach in falsifying the design assumptions and highlighting the causes of assumption violations.

Figures (18)

• Figure 1: Structure of a CPS, where a cyber component (green dashed box) interacts with a physical component (purple dashed box). In the cyber component, we highlight the controllers that implement the control layer and handle the low-level interaction with the physics. The control layer receives desired values for given physical quantities (the references) and uses sensors and actuators in order to enforce these values in the physical component. The measures and actuation signals create a closed-loop interaction between the control layer and the physical layer. In fact, CPS requirements are generally defined over quantities that live in the physical part of the system, captured by the CPS outputs.
• Figure 2: Simplified high-level description of the control system development; the red dashed arrows represent development steps, while the black solid arrows represent data flow. The development flow is simplified (e.g., neglecting iterations) to focus on the role of control engineering.
• Figure 3: Examples of DFT spectra (on the right) of different signals defined in the time-domain (on the left). A constant signal is described by only a zero frequency component, a pure sinusoidal maps to one single frequency component, and a non-periodic step maps to multiple frequencies.
• Figure 4: Example of how we can expect the CPS output $y$ (solid line) to track the desired value $r$ (dashed line). In the figure we intuitively highlight how a control system usually behaves like a low-pass filter, by filtering the fast-changing components of the input and tracking the slow-changing ones.
• Figure 5: Qualitative frequency-amplitude characterisation of the input space of a control loop. The colours highlight the validity of the linearised model with respect to the input frequency content and amplitude. The green area corresponds to the input signals for which the system remains within the assumptions of control theory. The azure area corresponds to the input signals that trigger non-linear phenomena but not enough to cause significant performance degradation in the system. The purple area corresponds to input signals for which control theory assumptions are not fulfilled and the behaviour of the system becomes unpredictable. We highlight the design scope (i.e., the union of the green and azure areas) with a dashed box.