A Roadmap for Simulation-Based Testing of Autonomous Cyber-Physical Systems: Challenges and Future Direction

TL;DR

The paper tackles the problem that traditional software testing is insufficient for autonomous cyber-physical systems (ACPS) by proposing a roadmap for simulation-based testing. It formalizes a test-case framework $\theta=(S,E,T,O)$ with $T=(a_0,...,a_n)$ to capture task sequences and oracle evaluation in simulators, and discusses the Reality Gap between simulated and real environments as well as the Oracle Problem. It identifies core challenges such as realism, benchmarking, and cost, and outlines automation and research directions—co-simulation, domain randomization, test minimization/selection/prioritization, and DevOps integration—to enable scalable, reproducible ACPS testing in simulation. The work aims to provide a unified framework and practical guidance for researchers and practitioners to develop safer, more reliable ACPS through standardized, cost-effective simulation-based testing and open benchmarks.

Abstract

As the era of autonomous cyber-physical systems (ACPSs), such as unmanned aerial vehicles and self-driving cars, unfolds, the demand for robust testing methodologies is key to realizing the adoption of such systems in real-world scenarios. However, traditional software testing paradigms face unprecedented challenges in ensuring the safety and reliability of these systems. In response, this paper pioneers a strategic roadmap for simulation-based testing of ACPSs, specifically focusing on autonomous systems. Our paper discusses the relevant challenges and obstacles of ACPSs, focusing on test automation and quality assurance, hence advocating for tailored solutions to address the unique demands of autonomous systems. While providing concrete definitions of test cases within simulation environments, we also accentuate the need to create new benchmark assets and the development of automated tools tailored explicitly for autonomous systems in the software engineering community. This paper not only highlights the relevant, pressing issues the software engineering community should focus on (in terms of practices, expected automation, and paradigms), but it also outlines ways to tackle them. By outlining the various domains and challenges of simulation-based testing/development for ACPSs, we provide directions for future research efforts.

Figures (1)

• Figure 1: The test subject $S$ is embedded in an environment $E$, from which we abstract many aspects away und only use a fraction as $E_{test}$.