CasModaTest: A Cascaded and Model-agnostic Self-directed Framework for Unit Test Generation
Chao Ni, Xiaoya Wang, Liushan Chen, Dehai Zhao, Zhengong Cai, Shaohua Wang, Xiaohu Yang
TL;DR
CasModaTest tackles limitations of ML-based unit test generation by introducing a cascaded, model-agnostic, end-to-end framework that separately generates a test prefix and a test oracle and then validates the complete test via assembling, compiling, and executing with automated repair. It leverages structured demonstration pools and carefully designed prompts to drive in-context learning, achieving substantial gains over SOTA baselines on Defects4J and demonstrating strong model-agnostic performance with open LLMs. The work emphasizes prefix-oracle alignment, two-stage generation, and demonstration selection as critical factors in producing executable, high-quality unit tests across diverse Java projects. The approach offers a practical, privacy-preserving pathway for automated testing that can adapt to both closed- and open-source LLMs.
Abstract
Though many machine learning (ML)-based unit testing generation approaches have been proposed and indeed achieved remarkable performance, they still have several limitations in effectiveness and practical usage. More precisely, existing ML-based approaches (1) generate partial content of a unit test, mainly focusing on test oracle generation; (2) mismatch the test prefix with the test oracle semantically; and (3) are highly bound with the close-sourced model, eventually damaging data security. We propose CasModaTest, a cascaded, model-agnostic, and end-to-end unit test generation framework, to alleviate the above limitations with two cascaded stages: test prefix generation and test oracle generation. Then, we manually build large-scale demo pools to provide CasModaTest with high-quality test prefixes and test oracles examples. Finally, CasModaTest automatically assembles the generated test prefixes and test oracles and compiles or executes them to check their effectiveness, optionally appending with several attempts to fix the errors occurring in compiling and executing phases. To evaluate the effectiveness of CasModaTest, we conduct large-scale experiments on a widely used dataset (Defects4J) and compare it with four state-of-the-art (SOTA) approaches by considering two performance measures. The experimental results indicate that CasModaTest outperforms all SOTAs with a substantial improvement (i.e., 60.62%-352.55% in terms of accuracy, 2.83%-87.27% in terms of focal method coverage). Besides, we also conduct experiments of CasModaTest on different open-source LLMs and find that CasModaTest can also achieve significant improvements over SOTAs (39.82%-293.96% and 9.25%-98.95% in terms of accuracy and focal method coverage, respectively) in end-to-end unit test generation