Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ChatUniTest: A Framework for LLM-Based Test Generation

Yinghao Chen, Zehao Hu, Chen Zhi, Junxiao Han, Shuiguang Deng, Jianwei Yin

TL;DR

<3-5 sentence high-level summary>ChatUniTest introduces an LLM-driven framework for automated unit test generation that addresses two key limitations of prior approaches: limited relevant context and insufficient validation. It combines an adaptive focal context mechanism with a generation-validation-repair cycle, supported by a reusable Core and a Toolchain of plugins. The approach integrates static analysis-derived dependencies, chain-of-thought prompting, and multiple validation stages (syntactic, compile, runtime) with rule-based and LLM-based repair to improve correctness and coverage. Empirical results across multiple Java projects show higher overall line coverage than EvoSuite and TestSpark, complemented by a user study indicating strong practical usefulness and adoption potential.

Abstract

Unit testing is an essential yet frequently arduous task. Various automated unit test generation tools have been introduced to mitigate this challenge. Notably, methods based on large language models (LLMs) have garnered considerable attention and exhibited promising results in recent years. Nevertheless, LLM-based tools encounter limitations in generating accurate unit tests. This paper presents ChatUniTest, an LLM-based automated unit test generation framework. ChatUniTest incorporates an adaptive focal context mechanism to encompass valuable context in prompts and adheres to a generation-validation-repair mechanism to rectify errors in generated unit tests. Subsequently, we have developed ChatUniTest Core, a common library that implements core workflow, complemented by the ChatUniTest Toolchain, a suite of seamlessly integrated tools enhancing the capabilities of ChatUniTest. Our effectiveness evaluation reveals that ChatUniTest outperforms TestSpark and EvoSuite in half of the evaluated projects, achieving the highest overall line coverage. Furthermore, insights from our user study affirm that ChatUniTest delivers substantial value to various stakeholders in the software testing domain. ChatUniTest is available at https://github.com/ZJU-ACES-ISE/ChatUniTest, and the demo video is available at https://www.youtube.com/watch?v=GmfxQUqm2ZQ.

ChatUniTest: A Framework for LLM-Based Test Generation

TL;DR

<3-5 sentence high-level summary>ChatUniTest introduces an LLM-driven framework for automated unit test generation that addresses two key limitations of prior approaches: limited relevant context and insufficient validation. It combines an adaptive focal context mechanism with a generation-validation-repair cycle, supported by a reusable Core and a Toolchain of plugins. The approach integrates static analysis-derived dependencies, chain-of-thought prompting, and multiple validation stages (syntactic, compile, runtime) with rule-based and LLM-based repair to improve correctness and coverage. Empirical results across multiple Java projects show higher overall line coverage than EvoSuite and TestSpark, complemented by a user study indicating strong practical usefulness and adoption potential.

Abstract

Unit testing is an essential yet frequently arduous task. Various automated unit test generation tools have been introduced to mitigate this challenge. Notably, methods based on large language models (LLMs) have garnered considerable attention and exhibited promising results in recent years. Nevertheless, LLM-based tools encounter limitations in generating accurate unit tests. This paper presents ChatUniTest, an LLM-based automated unit test generation framework. ChatUniTest incorporates an adaptive focal context mechanism to encompass valuable context in prompts and adheres to a generation-validation-repair mechanism to rectify errors in generated unit tests. Subsequently, we have developed ChatUniTest Core, a common library that implements core workflow, complemented by the ChatUniTest Toolchain, a suite of seamlessly integrated tools enhancing the capabilities of ChatUniTest. Our effectiveness evaluation reveals that ChatUniTest outperforms TestSpark and EvoSuite in half of the evaluated projects, achieving the highest overall line coverage. Furthermore, insights from our user study affirm that ChatUniTest delivers substantial value to various stakeholders in the software testing domain. ChatUniTest is available at https://github.com/ZJU-ACES-ISE/ChatUniTest, and the demo video is available at https://www.youtube.com/watch?v=GmfxQUqm2ZQ.
Paper Structure (23 sections, 3 figures, 1 table)

This paper contains 23 sections, 3 figures, 1 table.

Table of Contents

  1. introduction
  2. Related Work
  3. Approach
  4. Preparation
  5. Parsing
  6. Analysis
  7. Prompt Construction
  8. Adaptive Focal Context Generation
  9. Prompt Template
  10. Chain-of-thought
  11. Generation
  12. Syntactic, Compile and Runtime Validation
  13. Syntactic Validation
  14. Compile Validation
  15. Runtime Validation
  16. ...and 8 more sections

Figures (3)

  • Figure 1: overview of chatunitest-core and tools derived
  • Figure 2: Buggy method equals in Apache Commons.
  • Figure 3: The Test Case generated by ChatUniTest.