Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

How well LLM-based test generation techniques perform with newer LLM versions?

Michael Konstantinou, Renzo Degiovanni, Mike Papadakis

TL;DR

The paper evaluates whether newer LLM versions render prior engineered LLM-based test-generation techniques obsolete by replicating four state-of-the-art tools and comparing them to a plain-prompt baseline on 393 Java classes / 3,657 methods. Using modern LLMs, plain prompting achieves higher total line coverage, branch coverage, and mutation score than prior tools, suggesting a diminishing marginal benefit from complex prompt engineering. Granularity matters: method-level prompting yields more tests and higher coverage than class-level prompting, while a hybrid approach can maintain comparable effectiveness with substantially fewer LLM calls. Across multiple LLMs, results generalize, but a persistent challenge remains—the generation of non-compiling and failing tests, along with auxiliary content beyond unit tests—pointing to future directions in test repair, guidance, and multi-granularity strategies to improve cost-effectiveness and reliability.

Abstract

The rapid evolution of Large Language Models (LLMs) has strongly impacted software engineering, leading to a growing number of studies on automated unit test generation. However, the standalone use of LLMs without post-processing has proven insufficient, often producing tests that fail to compile or achieve high coverage. Several techniques have been proposed to address these issues, reporting improvements in test compilation and coverage. While important, LLM-based test generation techniques have been evaluated against relatively weak baselines (for todays' standards), i.e., old LLM versions and relatively weak prompts, which may exacerbate the performance contribution of the approaches. In other words, stronger (newer) LLMs may obviate any advantage these techniques bring. We investigate this issue by replicating four state-of-the-art LLM-based test generation tools, HITS, SymPrompt, TestSpark, and CoverUp that include engineering components aimed at guiding the test generation process through compilation and execution feedback, and evaluate their relative effectiveness and efficiency over a plain LLM test generation method. We integrate current LLM versions in all approaches and run an experiment on 393 classes and 3,657 methods. Our results show that the plain LLM approach can outperform previous state-of-the-art approaches in all test effectiveness metrics we used: line coverage (by 17.72%), branch coverage (by 19.80%) and mutation score (by 20.92%), and it does so at a comparable cost (LLM queries). We also observe that the granularity at which the plain LLM is applied has a significant impact on the cost. We therefore propose targeting first the program classes, where test generation is more efficient, and then the uncovered methods to reduce the number of LLM requests. This strategy achieves comparable (slightly higher) effectiveness while requiring about 20% fewer LLM requests.

How well LLM-based test generation techniques perform with newer LLM versions?

TL;DR

The paper evaluates whether newer LLM versions render prior engineered LLM-based test-generation techniques obsolete by replicating four state-of-the-art tools and comparing them to a plain-prompt baseline on 393 Java classes / 3,657 methods. Using modern LLMs, plain prompting achieves higher total line coverage, branch coverage, and mutation score than prior tools, suggesting a diminishing marginal benefit from complex prompt engineering. Granularity matters: method-level prompting yields more tests and higher coverage than class-level prompting, while a hybrid approach can maintain comparable effectiveness with substantially fewer LLM calls. Across multiple LLMs, results generalize, but a persistent challenge remains—the generation of non-compiling and failing tests, along with auxiliary content beyond unit tests—pointing to future directions in test repair, guidance, and multi-granularity strategies to improve cost-effectiveness and reliability.

Abstract

The rapid evolution of Large Language Models (LLMs) has strongly impacted software engineering, leading to a growing number of studies on automated unit test generation. However, the standalone use of LLMs without post-processing has proven insufficient, often producing tests that fail to compile or achieve high coverage. Several techniques have been proposed to address these issues, reporting improvements in test compilation and coverage. While important, LLM-based test generation techniques have been evaluated against relatively weak baselines (for todays' standards), i.e., old LLM versions and relatively weak prompts, which may exacerbate the performance contribution of the approaches. In other words, stronger (newer) LLMs may obviate any advantage these techniques bring. We investigate this issue by replicating four state-of-the-art LLM-based test generation tools, HITS, SymPrompt, TestSpark, and CoverUp that include engineering components aimed at guiding the test generation process through compilation and execution feedback, and evaluate their relative effectiveness and efficiency over a plain LLM test generation method. We integrate current LLM versions in all approaches and run an experiment on 393 classes and 3,657 methods. Our results show that the plain LLM approach can outperform previous state-of-the-art approaches in all test effectiveness metrics we used: line coverage (by 17.72%), branch coverage (by 19.80%) and mutation score (by 20.92%), and it does so at a comparable cost (LLM queries). We also observe that the granularity at which the plain LLM is applied has a significant impact on the cost. We therefore propose targeting first the program classes, where test generation is more efficient, and then the uncovered methods to reduce the number of LLM requests. This strategy achieves comparable (slightly higher) effectiveness while requiring about 20% fewer LLM requests.
Paper Structure (32 sections, 2 equations, 5 figures, 8 tables)

This paper contains 32 sections, 2 equations, 5 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Research Questions
  4. RQ1: What is the relative effectiveness of LLM-based test generation techniques when using recent LLMs?
  5. RQ2: What is the relative efficiency (cost) of these LLM-based test generation techniques?
  6. RQ3: How the granularity level (Class vs Method) affect the effectiveness and cost of the LLM-based test generation techniques?
  7. Experimental design
  8. Projects used
  9. Metrics used
  10. LLM-based unit test generators
  11. LLM-Plain
  12. Class-level workflow
  13. Method-level workflow
  14. Results
  15. RQ1: Effectiveness
  16. ...and 17 more sections

Figures (5)

  • Figure 1: LLM-Plain unit test generation workflow
  • Figure 2: Number of API requests between different approaches
  • Figure 3: Number of API requests for LLM-Plain across multiple LLMs
  • Figure 4: Additional content generated by LLMs instead of fixing unit tests
  • Figure 5: Average additional content generated on whole dataset