Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

EvoGPT: Leveraging LLM-Driven Seed Diversity to Improve Search-Based Test Suite Generation

Lior Broide, Roni Stern, Argaman Mordoch

TL;DR

EvoGPT addresses the limited diversity and premature convergence of traditional SBST by combining diverse, prompt- and temperature-varied LLM-generated test populations with an evolutionary algorithm. It introduces a generation-repair loop and coverage-guided augmentation during initialization, plus a plateau-escape mechanism that injects LLM-generated tests when progress stagnates. Empirical results on Defects4J show EvoGPT achieving higher line and branch coverage and mutation scores than both EvoSuite and TestART, with ablations confirming the critical role of diversity and plateau recovery. The work demonstrates that deliberate semantically diverse LLM injections, synchronized with EA optimization, yield substantial test-generation gains and suggests cost-aware avenues for practical deployment in real-world software testing.

Abstract

Search-Based Software Testing (SBST) is a well-established approach for automated unit test generation, yet it often suffers from premature convergence and limited diversity in the generated test suites. Recently, Large Language Models (LLMs) have emerged as an alternative technique for unit test generation. We present EvoGPT, a hybrid test generation system that integrates LLM-based test generation with SBST-based test suite optimization. EvoGPT uses LLMs to generate an initial population of test suites, and uses an Evolutionary Algorithm (EA) to further optimize this test suite population. A distinguishing feature of EvoGPT is its explicit enforcement of diversity, achieved through the use of multiple temperatures and prompt instructions during test generation. In addition, each LLM-generated test is refined using a generation-repair loop and coverage-guided assertion generation. To address evolutionary plateaus, EvoGPT also detects stagnation during search and injects additional LLM-generated tests aimed at previously uncovered branches. Here too diversity is enforced using multiple temperatures and prompt instructions. We evaluate EvoGPT on Defects4J, a standard benchmark for test generation. The results show that EvoGPT achieves, on average, a 10\% improvement in both code coverage and mutation score metrics compared to TestART, an LLM-only baseline; and EvoSuite, a standard SBST baseline. An ablation study indicates that explicitly enforcing diversity both at initialization and during the search is key to effectively leveraging LLMs for automated unit test generation.

EvoGPT: Leveraging LLM-Driven Seed Diversity to Improve Search-Based Test Suite Generation

TL;DR

EvoGPT addresses the limited diversity and premature convergence of traditional SBST by combining diverse, prompt- and temperature-varied LLM-generated test populations with an evolutionary algorithm. It introduces a generation-repair loop and coverage-guided augmentation during initialization, plus a plateau-escape mechanism that injects LLM-generated tests when progress stagnates. Empirical results on Defects4J show EvoGPT achieving higher line and branch coverage and mutation scores than both EvoSuite and TestART, with ablations confirming the critical role of diversity and plateau recovery. The work demonstrates that deliberate semantically diverse LLM injections, synchronized with EA optimization, yield substantial test-generation gains and suggests cost-aware avenues for practical deployment in real-world software testing.

Abstract

Search-Based Software Testing (SBST) is a well-established approach for automated unit test generation, yet it often suffers from premature convergence and limited diversity in the generated test suites. Recently, Large Language Models (LLMs) have emerged as an alternative technique for unit test generation. We present EvoGPT, a hybrid test generation system that integrates LLM-based test generation with SBST-based test suite optimization. EvoGPT uses LLMs to generate an initial population of test suites, and uses an Evolutionary Algorithm (EA) to further optimize this test suite population. A distinguishing feature of EvoGPT is its explicit enforcement of diversity, achieved through the use of multiple temperatures and prompt instructions during test generation. In addition, each LLM-generated test is refined using a generation-repair loop and coverage-guided assertion generation. To address evolutionary plateaus, EvoGPT also detects stagnation during search and injects additional LLM-generated tests aimed at previously uncovered branches. Here too diversity is enforced using multiple temperatures and prompt instructions. We evaluate EvoGPT on Defects4J, a standard benchmark for test generation. The results show that EvoGPT achieves, on average, a 10\% improvement in both code coverage and mutation score metrics compared to TestART, an LLM-only baseline; and EvoSuite, a standard SBST baseline. An ablation study indicates that explicitly enforcing diversity both at initialization and during the search is key to effectively leveraging LLMs for automated unit test generation.
Paper Structure (31 sections, 1 figure, 7 tables, 1 algorithm)

This paper contains 31 sections, 1 figure, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background and Problem Definition
  3. Traditional Test Generation Methods.
  4. LLM-based test generation methods.
  5. LLMs and Evolutionary Algorithms
  6. LLMs and EAs for Automated Test Generation
  7. The EvoGPT System
  8. Initial Population Generation
  9. Generation-Repair Loop
  10. Enhancing Coverage
  11. Population Optimization
  12. Fitness Score
  13. Selection
  14. Crossover
  15. Mutation
  16. ...and 16 more sections

Figures (1)

  • Figure 1: Overview of the EvoGPT system, illustrating the initial population generation process and the population optimization process. The former includes diverse system prompts and temperatures, a generation-repair loop, and a coverage enhancement step. The latter includes an evolutionary algorithm, search plateau monitoring, and an LLM-based plateau escaping test generation step that also uses diverse system prompts and temperatures.