Automated Control Logic Test Case Generation using Large Language Models
Heiko Koziolek, Virendra Ashiwal, Soumyadip Bandyopadhyay, Chandrika K R
TL;DR
The paper tackles the challenge of labor-intensive PLC/DCS test-case generation by introducing an automated method that leverages Large Language Models to synthesize test inputs and expected outputs from IEC 61131-3 Structured Text code. A CSV-based output from the LLM is converted into executable ST-code, transpiled to C with MATIEC, and run in a soft-PLC environment to collect coverage and assertion data, enabling rapid evaluation. In experiments on ten OSCAT function blocks, the approach achieves high statement coverage for simple-to-medium blocks, with enhanced prompts increasing input coverage but often producing incorrect assertions that require manual adjustment. The work demonstrates that LLM-based test generation can save engineers time and serve as a complementary component to symbolic or search-based methods, while highlighting the need for improved assertion reliability and timer support for more complex PLC code.
Abstract
Testing PLC and DCS control logic in industrial automation is laborious and challenging since appropriate test cases are often complex and difficult to formulate. Researchers have previously proposed several automated test case generation approaches for PLC software applying symbolic execution and search-based techniques. Often requiring formal specifications and performing a mechanical analysis of programs, these approaches may uncover specific programming errors but sometimes suffer from state space explosion and cannot process rather informal specifications. We proposed a novel approach for the automatic generation of PLC test cases that queries a Large Language Model (LLM) to synthesize test cases for code provided in a prompt. Experiments with ten open-source function blocks from the OSCAT automation library showed that the approach is fast, easy to use, and can yield test cases with high statement coverage for low-to-medium complex programs. However, we also found that LLM-generated test cases suffer from erroneous assertions in many cases, which still require manual adaption.