Tracking the Moving Target: A Framework for Continuous Evaluation of LLM Test Generation in Industry

TL;DR

The paper tackles the challenge of rapidly evolving LLMs used for automated test generation in industrial DevSecOps settings, where point-in-time evaluations quickly become obsolete. It proposes a continuous measurement framework that integrates with industry tools like SonarQube to assess both technical adequacy (e.g., compilation, code coverage) and practical qualities (maintainability, expert assessment) through a structured methodology and prompt engineering. The authors validate the framework via a longitudinal study at LKS Next, showing substantial improvements in code quality and test coverage from March 2024 to December 2024 as LLMs evolved, while highlighting persistent needs for expert oversight and cost/privacy considerations. The findings offer practical guidance for organizations seeking to adopt LLM-based test generation, emphasizing ongoing evaluation, careful model selection, and disciplined prompt design to achieve industrial viability. Overall, the work bridges academic evaluation with real-world deployment, advancing how companies can systematically track and improve LLM-assisted testing in evolving software pipelines.

Abstract

Large Language Models (LLMs) have shown great potential in automating software testing tasks, including test generation. However, their rapid evolution poses a critical challenge for companies implementing DevSecOps - evaluations of their effectiveness quickly become outdated, making it difficult to assess their reliability for production use. While academic research has extensively studied LLM-based test generation, evaluations typically provide point-in-time analyses using academic benchmarks. Such evaluations do not address the practical needs of companies who must continuously assess tool reliability and integration with existing development practices. This work presents a measurement framework for the continuous evaluation of commercial LLM test generators in industrial environments. We demonstrate its effectiveness through a longitudinal study at LKS Next. The framework integrates with industry-standard tools like SonarQube and provides metrics that evaluate both technical adequacy (e.g., test coverage) and practical considerations (e.g., maintainability or expert assessment). Our methodology incorporates strategies for test case selection, prompt engineering, and measurement infrastructure, addressing challenges such as data leakage and reproducibility. Results highlight both the rapid evolution of LLM capabilities and critical factors for successful industrial adoption, offering practical guidance for companies seeking to integrate these technologies into their development pipelines.

Figures (1)

• Figure 1: BPMN diagram showcasing the iterative measurement process.