Generative AI in Simulation-Based Test Environments for Large-Scale Cyber-Physical Systems: An Industrial Study
Masoud Sadrnezhaad, José Antonio Hernández López, Torvald Mårtensson, Daniel Varro
TL;DR
The paper investigates the potential of generative AI in simulation-based testing for large-scale cyber-physical systems by gathering industry practitioners' perspectives in a cross-company workshop. It presents a three-phase methodology (literature grounding, an in-depth workshop, and thematic coding) and reports concrete challenges around data scale, scenario evaluation, and pipeline integration. The authors propose a three-pronged research agenda—AI-generated scenarios and environment models, simulators integrated into CI/CD pipelines, and trustworthiness in AI for simulation—grounded in practitioner insights. The work highlights both the promise of AI to boost efficiency and coverage and the practical barriers that must be addressed through academia–industry collaboration and rigorous evaluation.
Abstract
Quality assurance for large-scale cyber-physical systems relies on sophisticated test activities using complex test environments investigated with the help of numerous types of simulators. As these systems grow, extensive resources are required to develop and maintain simulation models of hardware and software components, as well as physical environments. Meanwhile, recent advances in generative AI have led to tools that can produce executable test cases for software systems, offering potential benefits such as reducing manual efforts or increasing test coverage. However, the application of generative AI techniques to simulation-based testing of large-scale cyber-physical systems remains underexplored. To better understand this gap, this study captures practitioners' perspectives on leveraging generative AI, based on a cross-company workshop with six organizations. Our contribution is twofold: (1) detailed, experience-based insights into challenges faced by engineers, and (2) a research agenda comprising three high-priority directions: (a) AI-generated scenarios and environment models, (b) simulators and AI in CI/CD pipelines, and (c) trustworthiness in generative AI for simulation. While participants acknowledged substantial potential, they also highlighted unresolved challenges. By detailing these issues, the paper aims to guide future academia-industry collaboration towards the responsible adoption of generative AI in simulation-based testing.