Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mutation Testing via Iterative Large Language Model-Driven Scientific Debugging

Philipp Straubinger, Marvin Kreis, Stephan Lukasczyk, Gordon Fraser

TL;DR

This work investigates whether Scientific Debugging guided by Large Language Models can improve automated test generation for mutation testing in Python. By iteratively formulating hypotheses, running targeted experiments, and refining tests, the LLM-based approach achieves higher branch coverage and mutation scores than a state-of-the-art search-based tool (Pynguin), though at a higher computational cost. The study demonstrates that iterative prompting is a key driver of quality, while equivalence detection by LLMs remains challenging and often overestimates true equivalence. The findings suggest that LLM-driven, iterative test generation can meaningfully enhance fault detection and coverage, with practical implications for automated software testing workflows and future work toward broader language support and hybrid methods.

Abstract

Large Language Models (LLMs) can generate plausible test code. Intuitively they generate this by imitating tests seen in their training data, rather than reasoning about execution semantics. However, such reasoning is important when applying mutation testing, where individual tests need to demonstrate differences in program behavior between a program and specific artificial defects (mutants). In this paper, we evaluate whether Scientific Debugging, which has been shown to help LLMs when debugging, can also help them to generate tests for mutants. In the resulting approach, LLMs form hypotheses about how to kill specific mutants, and then iteratively generate and refine tests until they succeed, all with detailed explanations for each step. We compare this method to three baselines: (1) directly asking the LLM to generate tests, (2) repeatedly querying the LLM when tests fail, and (3) search-based test generation with Pynguin. Our experiments evaluate these methods based on several factors, including mutation score, code coverage, success rate, and the ability to identify equivalent mutants. The results demonstrate that LLMs, although requiring higher computation cost, consistently outperform Pynguin in generating tests with better fault detection and coverage. Importantly, we observe that the iterative refinement of test cases is important for achieving high-quality test suites.

Mutation Testing via Iterative Large Language Model-Driven Scientific Debugging

TL;DR

This work investigates whether Scientific Debugging guided by Large Language Models can improve automated test generation for mutation testing in Python. By iteratively formulating hypotheses, running targeted experiments, and refining tests, the LLM-based approach achieves higher branch coverage and mutation scores than a state-of-the-art search-based tool (Pynguin), though at a higher computational cost. The study demonstrates that iterative prompting is a key driver of quality, while equivalence detection by LLMs remains challenging and often overestimates true equivalence. The findings suggest that LLM-driven, iterative test generation can meaningfully enhance fault detection and coverage, with practical implications for automated software testing workflows and future work toward broader language support and hybrid methods.

Abstract

Large Language Models (LLMs) can generate plausible test code. Intuitively they generate this by imitating tests seen in their training data, rather than reasoning about execution semantics. However, such reasoning is important when applying mutation testing, where individual tests need to demonstrate differences in program behavior between a program and specific artificial defects (mutants). In this paper, we evaluate whether Scientific Debugging, which has been shown to help LLMs when debugging, can also help them to generate tests for mutants. In the resulting approach, LLMs form hypotheses about how to kill specific mutants, and then iteratively generate and refine tests until they succeed, all with detailed explanations for each step. We compare this method to three baselines: (1) directly asking the LLM to generate tests, (2) repeatedly querying the LLM when tests fail, and (3) search-based test generation with Pynguin. Our experiments evaluate these methods based on several factors, including mutation score, code coverage, success rate, and the ability to identify equivalent mutants. The results demonstrate that LLMs, although requiring higher computation cost, consistently outperform Pynguin in generating tests with better fault detection and coverage. Importantly, we observe that the iterative refinement of test cases is important for achieving high-quality test suites.

Paper Structure

This paper contains 37 sections, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background
  3. Large Language Models
  4. Mutation Testing
  5. Scientific Debugging
  6. Search-based Test Generation
  7. Scientific Test Generation
  8. Scientific Debugging Process
  9. Initial Prompting
  10. Hypothesis Formulation
  11. Experimentation
  12. Conclusions
  13. Test Cases
  14. Handling Equivalent Mutants
  15. Key Design Principles
  16. ...and 22 more sections

Figures (4)

  • Figure 1: Performance differences, each time along with the individual Pynguin test suites and the combined test suites.
  • Figure 2: Costs and tokens per mutants divided into (1) prompt tokens (cached), (2) prompt tokens (uncached), and (3) completion tokens.
  • Figure 3: Cost per project and approach.
  • Figure 4: Number of iterations and outcomes of our approaches.