Test Case Generation for Simulink Models: An Experience from the E-Bike Domain

TL;DR

The paper addresses the need for empirical evidence on the effectiveness of search-based software testing (SBST) for Simulink-based cyber-physical systems by applying the HECATE SBST framework to two e-Bike motor controllers across three requirements and six parameterized test sequences, totaling 36 experiments. It reports that HECATE found failure-revealing test cases in about 83% of runs (30/36), with PWM controller tests consistently exposing issues across all requirements and Buck tests mainly revealing regulatory and safety violations, as confirmed by domain experts. The study also provides detailed efficiency data, noting an average test-generation time of about 1 hour 17 minutes, with per-case simulation times and longer durations for certain Buck scenarios. These results offer practical lessons for industrial adoption, demonstrate the replicability of SBST in an industrial CPS context, and support generalization to similar domains while acknowledging domain-specific limitations.

Abstract

Cyber-physical systems development often requires engineers to search for defects in their Simulink models. Search-based software testing (SBST) is a standard technology that supports this activity. To increase practical adaption, industries need empirical evidence of the effectiveness and efficiency of (existing) SBST techniques on benchmarks from different domains and of varying complexity. To address this industrial need, this paper presents our experience assessing the effectiveness and efficiency of SBST in generating failure-revealing test cases for cyber-physical systems requirements. Our study subject is within the electric bike (e-Bike) domain and concerns the software controller of an e-Bike motor, particularly its functional, regulatory, and safety requirements. We assessed the effectiveness and efficiency of HECATE, an SBST framework for Simulink models, to analyze two software controllers. HECATE successfully identified failure-revealing test cases for 83% (30 out of 36) of our experiments. It required, on average, 1 h 17 min 26 s (min = 11 min 56 s, max = 8 h 16 min 22 s, std = 1 h 50 min 34 s) to compute the failure-revealing test cases. The developer of the e-Bike model confirmed the failures identified by HECATE. We present the lessons learned and discuss the relevance of our results for industrial applications, the state of practice improvement, and the results' generalizability.

Figures (11)

• Figure 1: Simulink® model for the e-Bike.
• Figure 2: Two software controllers for the e-Bike.
• Figure 3: Test Blocks for our e-Bike model.
• Figure 4: Signal types generated as input by HECATE for our e-Bike models.
• Figure 5: Parameterized Test Sequences.