Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Coverage Explorer: Coverage-guided Test Generation for Cyber Physical Systems

Sanaz Sheikhi, Stanley Bak

TL;DR

This study introduces a testing framework designed to systematically formulate test cases, effectively exploring the state space of CPS, which introduces a coverage-centric sampling technique, coupled with a cluster-based methodology for training a surrogate model.

Abstract

Given the safety-critical functions of autonomous cyber-physical systems (CPS) across diverse domains, testing these systems is essential. While conventional software and hardware testing methodologies offer partial insights, they frequently do not provide adequate coverage in a CPS. In this study, we introduce a testing framework designed to systematically formulate test cases, effectively exploring the state space of CPS. This framework introduces a coverage-centric sampling technique, coupled with a cluster-based methodology for training a surrogate model. The framework then uses model predictive control within the surrogate model to generates test cases tailored to CPS specifications. To evaluate the efficacy of the framework, we applied it on several benchmarks, spanning from a kinematic car to systems like an unmanned aircraft collision avoidance system (ACAS XU) and automatic transmission system. Comparative analyses were conducted against alternative test generation strategies, including randomized testing, as well as falsification using S-TaLiRo.

Coverage Explorer: Coverage-guided Test Generation for Cyber Physical Systems

TL;DR

This study introduces a testing framework designed to systematically formulate test cases, effectively exploring the state space of CPS, which introduces a coverage-centric sampling technique, coupled with a cluster-based methodology for training a surrogate model.

Abstract

Given the safety-critical functions of autonomous cyber-physical systems (CPS) across diverse domains, testing these systems is essential. While conventional software and hardware testing methodologies offer partial insights, they frequently do not provide adequate coverage in a CPS. In this study, we introduce a testing framework designed to systematically formulate test cases, effectively exploring the state space of CPS. This framework introduces a coverage-centric sampling technique, coupled with a cluster-based methodology for training a surrogate model. The framework then uses model predictive control within the surrogate model to generates test cases tailored to CPS specifications. To evaluate the efficacy of the framework, we applied it on several benchmarks, spanning from a kinematic car to systems like an unmanned aircraft collision avoidance system (ACAS XU) and automatic transmission system. Comparative analyses were conducted against alternative test generation strategies, including randomized testing, as well as falsification using S-TaLiRo.
Paper Structure (20 sections, 4 equations, 7 figures, 3 tables, 3 algorithms)

This paper contains 20 sections, 4 equations, 7 figures, 3 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. RELATED WORK
  3. Preliminaries
  4. CPS COVERAGE SCORE
  5. Koopman Operator Linearization
  6. Problem Statement
  7. METHODOLOGY
  8. Training Data Refinement
  9. Model Training
  10. Identifying state space boundaries
  11. Coverage-guided Sampling
  12. EVALUATION
  13. BENCHMARKS
  14. Evaluation of the coverage-guided model training
  15. Evaluation of training data boundary identification technique
  16. ...and 5 more sections

Figures (7)

  • Figure 1: Our proposed CPS coverage metric increases as more interesting events are generated, although events close in the objective space do not increase the score as much as separate events.
  • Figure 2: State space coverage
  • Figure 3: Coverage Explorer Overview
  • Figure 4: Coverage-guided sampling
  • Figure 5: Visual representation of enhanced training data for kinematic car benchmark, expanding coverage across the state space through model training iterations. Additionally, illustrating clustered similar state trajectories aiding in selecting representative data and helping to establish precise bounds for target sampling.
  • ...and 2 more figures