E-Test: E'er-Improving Test Suites
Ketai Qiu, Luca Di Grazia, Leonardo Mariani, Mauro Pezzè
TL;DR
E-Test addresses the persistent gap between production executions and the coverage provided by test suites by mining execution scenarios from production and classifying them into already-tested, need-test, and error-prone. The approach combines a three-phase pipeline (PreProcessor, Analyzer, PostProcessor) with a fine-tuned LLM and Retrieval-Augmented Generation to predict the testing value of new scenarios and automatically generate test cases, focusing on unit testing in this work. Evaluation on 1,975 scenarios from Defects4J and popular Java projects shows that E-Test achieves a higher F1-score (≈0.55) for identifying not-yet-tested scenarios than state-of-the-art baselines, and that generated tests reveal a substantial portion of defects (≈83%). The results demonstrate that production-driven, LLM-based test augmentation can significantly improve test-suite quality while reducing manual effort, with potential extension to integration and system testing using longer context models and telemetry data.
Abstract
Test suites are inherently imperfect, and testers can always enrich a suite with new test cases that improve its quality and, consequently, the reliability of the target software system. However, finding test cases that explore execution scenarios beyond the scope of an existing suite can be extremely challenging and labor-intensive, particularly when managing large test suites over extended periods. In this paper, we propose E-Test, an approach that reduces the gap between the execution space explored with a test suite and the executions experienced after testing by augmenting the test suite with test cases that explore execution scenarios that emerge in production. E-Test (i) identifies executions that have not yet been tested from large sets of scenarios, such as those monitored during intensive production usage, and (ii) generates new test cases that enhance the test suite. E-Test leverages Large Language Models (LLMs) to pinpoint scenarios that the current test suite does not adequately cover, and augments the suite with test cases that execute these scenarios. Our evaluation on a dataset of 1,975 scenarios, collected from highly-starred open-source Java projects already in production and Defects4J, demonstrates that E-Test retrieves not-yet-tested execution scenarios significantly better than state-of-the-art approaches. While existing regression testing and field testing approaches for this task achieve a maximum F1-score of 0.34, and vanilla LLMs achieve a maximum F1-score of 0.39, E-Test reaches 0.55. These results highlight the impact of E-Test in enhancing test suites by effectively targeting not-yet-tested execution scenarios and reducing manual effort required for maintaining test suites.