KTester: Leveraging Domain and Testing Knowledge for More Effective LLM-based Test Generation

TL;DR

This paper tackles the problem of unreliable LLM-based unit test generation by introducing KTester, a knowledge-aware framework that injects project-specific information and testing-domain knowledge into the generation process. It combines offline knowledge extraction (project structure, usage, and a function-level index) with an online five-step pipeline (framework generation, multi-perspective test-case design, test-method transformation, integration, and refinement) to produce semantically rich, maintainable test classes. Evaluated on the HITS dataset, KTester outperforms state-of-the-art baselines across multiple correctness and coverage metrics (e.g., CPR of 100% and LC around 62.8%), while requiring fewer test cases and yielding higher-quality test code according to a human study. The results validate the practical value of matching LLM prompts and generation workflows to codebase context and testing principles, and point to broader applicability to other languages and dynamic analysis enhancements in future work.

Abstract

Automated unit test generation using large language models (LLMs) holds great promise but often struggles with generating tests that are both correct and maintainable in real-world projects. This paper presents KTester, a novel framework that integrates project-specific knowledge and testing domain knowledge to enhance LLM-based test generation. Our approach first extracts project structure and usage knowledge through static analysis, which provides rich context for the model. It then employs a testing-domain-knowledge-guided separation of test case design and test method generation, combined with a multi-perspective prompting strategy that guides the LLM to consider diverse testing heuristics. The generated tests follow structured templates, improving clarity and maintainability. We evaluate KTester on multiple open-source projects, comparing it against state-of-the-art LLM-based baselines using automatic correctness and coverage metrics, as well as a human study assessing readability and maintainability. Results demonstrate that KTester significantly outperforms existing methods across six key metrics, improving execution pass rate by 5.69% and line coverage by 8.83% over the strongest baseline, while requiring less time and generating fewer test cases. Human evaluators also rate the tests produced by KTester significantly higher in terms of correctness, readability, and maintainability, confirming the practical advantages of our knowledge-driven framework.

Figures (11)

• Figure 1: Motivational Examples(a): Setting Tested Object Incorrectly
• Figure 2: Motivational Examples(b): Insufficient Assert Statements
• Figure 3: Motivational Examples(c): Hard Encoding Values
• Figure 4: The Framework of KTester
• Figure 5: Test Class Example