Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An Empirical Evaluation of White-box and Black-box Test Case Prioritization Techniques in CPSs Modeled in Simulink

Aitor Arrieta

TL;DR

This paper tackles the high cost of regression testing for CPSs modeled in MATLAB/Simulink and compares white-box and black-box test case prioritization techniques in this domain. It conducts the largest empirical study to date, evaluating 11 prioritization techniques across six Simulink subject models, using APFD as the fault-detection metric and 100 runs with non-parametric statistics. Key findings show that white-box methods are generally competitive with, and often slightly superior to, black-box methods, with total-DC excelling on larger models, and anti-pattern–based black-box metrics performing well except in a few cases like Swimairspeed; black-box methods also run much faster. The work provides actionable guidance for practitioners, plus public implementations and datasets to support replication and deployment in CPS regression testing.

Abstract

MATLAB/Simulink is the leading tool for simulating complex Cyber-Physical Systems (CPSs). The simulation models of complex CPSs are typically compute intensive, and the execution of test cases is long. Furthermore, the execution of test cases is typically triggered several times at different ``in-the-Loop'' test levels (i.e., Model, Software and Hardware-in-the-Loop). Therefore, test optimization techniques, such as test case prioritization, are paramount when testing these systems. In this paper, we present the largest empirical study on test case prioritization techniques for Simulink models by comparing the performance of white-box and black-box test case prioritization techniques. We assess traditional test case prioritization techniques, and we also propose new approaches for use in the context of Simulink models. We empirically compared 11 test case prioritization techniques using six Simulink models of different sizes and complexities. When comparing white-box against black-box test case prioritization techniques, we found that in general, white-box techniques were slightly better than black-box ones. In the context of white-box test case prioritization, the total greedy approach performed better than the additional greedy techniques in larger models. As for the test case prioritization time, black-box techniques were faster, although total greedy techniques were fast enough to be used in practice.

An Empirical Evaluation of White-box and Black-box Test Case Prioritization Techniques in CPSs Modeled in Simulink

TL;DR

This paper tackles the high cost of regression testing for CPSs modeled in MATLAB/Simulink and compares white-box and black-box test case prioritization techniques in this domain. It conducts the largest empirical study to date, evaluating 11 prioritization techniques across six Simulink subject models, using APFD as the fault-detection metric and 100 runs with non-parametric statistics. Key findings show that white-box methods are generally competitive with, and often slightly superior to, black-box methods, with total-DC excelling on larger models, and anti-pattern–based black-box metrics performing well except in a few cases like Swimairspeed; black-box methods also run much faster. The work provides actionable guidance for practitioners, plus public implementations and datasets to support replication and deployment in CPS regression testing.

Abstract

MATLAB/Simulink is the leading tool for simulating complex Cyber-Physical Systems (CPSs). The simulation models of complex CPSs are typically compute intensive, and the execution of test cases is long. Furthermore, the execution of test cases is typically triggered several times at different ``in-the-Loop'' test levels (i.e., Model, Software and Hardware-in-the-Loop). Therefore, test optimization techniques, such as test case prioritization, are paramount when testing these systems. In this paper, we present the largest empirical study on test case prioritization techniques for Simulink models by comparing the performance of white-box and black-box test case prioritization techniques. We assess traditional test case prioritization techniques, and we also propose new approaches for use in the context of Simulink models. We empirically compared 11 test case prioritization techniques using six Simulink models of different sizes and complexities. When comparing white-box against black-box test case prioritization techniques, we found that in general, white-box techniques were slightly better than black-box ones. In the context of white-box test case prioritization, the total greedy approach performed better than the additional greedy techniques in larger models. As for the test case prioritization time, black-box techniques were faster, although total greedy techniques were fast enough to be used in practice.

Paper Structure

This paper contains 29 sections, 10 equations, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Simulation models and basic notation
  4. Simulation-based testing
  5. Selected Test Case Prioritization Methods
  6. Black-Box Techniques
  7. Techniques based on anti-patterns
  8. Techniques based on test similarity metrics
  9. White-Box Techniques
  10. Test case prioritization techniques not selected in this study
  11. Empirical Evaluation
  12. Research Questions
  13. Selected Benchmark
  14. Subject Models
  15. Test Cases
  16. ...and 14 more sections

Figures (4)

  • Figure 1: Example of a simulation model of a Cruise Controller of a car Arrieta2019a
  • Figure 2: An example of the execution of a test case in the EMB case study system
  • Figure 3: Distribution of the obtained APFD values for the six subject models and the selected techniques
  • Figure 4: An example showing the problems of the Euclidean distance at capturing signal shapes