LLM-based Unit Test Generation via Property Retrieval

TL;DR

Property-Based Retrieval Augmentation is proposed, a novel mechanism that extends LLM-based Retrieval-Augmented Generation beyond basic vector, text similarity, and graph-based methods and introduces a novel code-context-aware retrieval mechanism for LLMs beyond general context, offering valuable insights and potential applications for other code-related tasks.

Abstract

Automated unit test generation has been widely studied, with Large Language Models (LLMs) recently showing significant potential. Moreover, in the context of unit test generation, these tools prioritize high code coverage, often at the expense of practical usability, correctness, and maintainability. In response, we propose Property-Based Retrieval Augmentation, a novel mechanism that extends LLM-based Retrieval-Augmented Generation (RAG) beyond basic vector, text similarity, and graph-based methods. Our approach considers task-specific context and introduces a tailored property retrieval mechanism. Specifically, in the unit test generation task, we account for the unique structure of unit tests by dividing the test generation process into Given, When, and Then phases. When generating tests for a focal method, we not only retrieve general context for the code under test but also consider task-specific context such as pre-existing tests of other methods, which can provide valuable insights for any of the Given, When, and Then phases. This forms property relationships between focal method and other methods, thereby expanding the scope of retrieval beyond traditional RAG. We implement this approach in a tool called APT, which sequentially performs preprocessing, property retrieval, and unit test generation, using an iterative strategy where newly generated tests guide the creation of subsequent ones. We evaluated APT on 12 open-source projects with 1515 methods, and the results demonstrate that APT consistently outperforms existing tools in terms of correctness, completeness, and maintainability of the generated tests. Moreover, we introduce a novel code-context-aware retrieval mechanism for LLMs beyond general context, offering valuable insights and potential applications for other code-related tasks.

Figures (12)

• Figure 1: The method to be tested from the RedissonLongAdder class: reset method
• Figure 2: The Unit Test for reset generated by GitHub Copilot
• Figure 3: Two examples of Redisson LongAdder methods demonstrating the reuse of patterns from the sum method's unit test to generate a unit test for the reset method. The figure illustrates the similarities in the initialization, invocation, and assertion phases of the sum and reset methods.
• Figure 4: RedissonAtomicDouble and RedissonAtomicLong both implementing the abstract class RedissonExpirable
• Figure 5: Unit tests for the getAndSet and getAndDelete methods in RedissonAtomicDouble and RedissonAtomicLong. The tests in RedissonAtomicDoubleTest can provide guidance for generating tests in RedissonAtomicLongTest due to the structural and behavioral similarities of the methods.