LLMs-Powered Real-Time Fault Injection: An Approach Toward Intelligent Fault Test Cases Generation

TL;DR

This work tackles the high cost and complexity of manual FI attribute specification in ISO 26262-compliant automotive safety testing by proposing an LLM-assisted framework that generates fault test cases (TCs) from functional safety requirements (FSRs) for real-time FI on hardware-in-the-loop (HIL) systems. It introduces a three-phase pipeline—LLM-driven TC generation from FSRs, real-time fault injection via RTICANMM, and result analysis—with FSRs classified into sensor- or actuator-related categories before TC generation, and a JSON output guiding fault location and timing within ISO 26262 constraints. The evaluation covers multiple LLMs (GPT-4o, GPT-4o-mini, phi4-14b, qwen2.5-7b, llama-3-70b), reporting high performance: a classification F1 of $88.0\%$ and TC generation F1 of $97.5\%$ for GPT-4o, and a strong open-source alternative with Phi-4-14b achieving near-parallel TC-generation performance ($96.6\%$ in conclusion). Real-time experiments on a high-fidelity gasoline engine and vehicle dynamics model demonstrate the approach’s ability to reveal FI-induced safety violations and support corrective actions, reducing testing costs while enhancing safety. The results indicate that LLM-driven FI can efficiently automate TC generation and enable scalable, real-time safety validation in complex ASSs.

Abstract

A well-known testing method for the safety evaluation and real-time validation of automotive software systems (ASSs) is Fault Injection (FI). In accordance with the ISO 26262 standard, the faults are introduced artificially for the purpose of analyzing the safety properties and verifying the safety mechanisms during the development phase. However, the current FI method and tools have a significant limitation in that they require manual identification of FI attributes, including fault type, location and time. The more complex the system, the more expensive, time-consuming and labour-intensive the process. To address the aforementioned challenge, a novel Large Language Models (LLMs)-assisted fault test cases (TCs) generation approach for utilization during real-time FI tests is proposed in this paper. To this end, considering the representativeness and coverage criteria, the applicability of various LLMs to create fault TCs from the functional safety requirements (FSRs) has been investigated. Through the validation results of LLMs, the superiority of the proposed approach utilizing gpt-4o in comparison to other state-of-the-art models has been demonstrated. Specifically, the proposed approach exhibits high performance in terms of FSRs classification and fault TCs generation with F1-score of 88% and 97.5%, respectively. To illustrate the proposed approach, the generated fault TCs were executed in real time on a hardware-in-the-loop system, where a high-fidelity automotive system model served as a case study. This novel approach offers a means of optimizing the real-time testing process, thereby reducing costs while simultaneously enhancing the safety properties of complex safety-critical ASSs.

Figures (8)

• Figure 1: Proposed LLMs-assisted real-time fault injection approach with HIL simulation.
• Figure 2: Real-time Hardware-in-the-Loop simulation system.
• Figure 3: Number of FSR per Sensor.
• Figure 4: Fault effect analysis during the delay fault occurrence in APP sensor.
• Figure 5: Engine speed behavior under concurrent faults.