Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Comprehensive Study on Large Language Models for Mutation Testing

Bo Wang, Mingda Chen, Ming Deng, Youfang Lin, Mark Harman, Mike Papadakis, Jie M. Zhang

TL;DR

The paper addresses the lack of large-scale empirical evaluation for LLM-based mutation testing by comparing BugFarm, LLMorpheus, and a novel LLMut prompt against traditional methods across 851 real Java bugs. It demonstrates that LLM-based approaches yield more diverse and fault-revealing mutants, achieving substantially higher Real Bug Detection Rates and Average Ochiai Coefficients, but at the cost of increased non-compilable, duplicate, and equivalent mutants and higher generation time. The authors introduce a high-quality, richly annotated dataset and a comprehensive prompt design, providing a robust baseline and practical guidance for deploying LLM-based mutation testing while highlighting key challenges to be mitigated. The work informs practitioners and researchers about the trade-offs between effectiveness and validity, and points to future directions such as high-order mutants, cross-language generalization, and hybrid techniques to harness LLM strengths while maintaining reliability.

Abstract

Large Language Models (LLMs) have recently been used to generate mutants in both research work and in industrial practice. However, there has been no comprehensive empirical study of their performance for this increasingly important LLM-based Software Engineering application. To address this, we conduct a comprehensive empirical study evaluating BugFarm and LLMorpheus (the two state-of-the-art LLM-based approaches), alongside seven LLMs using our newly designed prompt, including both leading open- and closed-source models, on 851 real bugs from two Java real-world bug benchmarks. Our results reveal that, compared to existing rule-based approaches, LLMs generate more diverse mutants, that are behaviorally closer to real bugs and, most importantly, with 111.29% higher fault detection. That is, 87.98% (for LLMs) vs. 41.64% (for rule-based); an increase of 46.34 percentage points. Nevertheless, our results also reveal that these impressive results for improved effectiveness come at a cost: the LLM-generated mutants have worse non-compilability, duplication, and equivalent mutant rates by 26.60, 10.14, and 3.51 percentage points, respectively. These findings are immediately actionable for both research and practice. They allow practitioners to have greater confidence in deploying LLM-based mutation, while researchers now have a baseline for the state-of-the-art, with which they can research techniques to further improve effectiveness and reduce cost.

A Comprehensive Study on Large Language Models for Mutation Testing

TL;DR

The paper addresses the lack of large-scale empirical evaluation for LLM-based mutation testing by comparing BugFarm, LLMorpheus, and a novel LLMut prompt against traditional methods across 851 real Java bugs. It demonstrates that LLM-based approaches yield more diverse and fault-revealing mutants, achieving substantially higher Real Bug Detection Rates and Average Ochiai Coefficients, but at the cost of increased non-compilable, duplicate, and equivalent mutants and higher generation time. The authors introduce a high-quality, richly annotated dataset and a comprehensive prompt design, providing a robust baseline and practical guidance for deploying LLM-based mutation testing while highlighting key challenges to be mitigated. The work informs practitioners and researchers about the trade-offs between effectiveness and validity, and points to future directions such as high-order mutants, cross-language generalization, and hybrid techniques to harness LLM strengths while maintaining reliability.

Abstract

Large Language Models (LLMs) have recently been used to generate mutants in both research work and in industrial practice. However, there has been no comprehensive empirical study of their performance for this increasingly important LLM-based Software Engineering application. To address this, we conduct a comprehensive empirical study evaluating BugFarm and LLMorpheus (the two state-of-the-art LLM-based approaches), alongside seven LLMs using our newly designed prompt, including both leading open- and closed-source models, on 851 real bugs from two Java real-world bug benchmarks. Our results reveal that, compared to existing rule-based approaches, LLMs generate more diverse mutants, that are behaviorally closer to real bugs and, most importantly, with 111.29% higher fault detection. That is, 87.98% (for LLMs) vs. 41.64% (for rule-based); an increase of 46.34 percentage points. Nevertheless, our results also reveal that these impressive results for improved effectiveness come at a cost: the LLM-generated mutants have worse non-compilability, duplication, and equivalent mutant rates by 26.60, 10.14, and 3.51 percentage points, respectively. These findings are immediately actionable for both research and practice. They allow practitioners to have greater confidence in deploying LLM-based mutation, while researchers now have a baseline for the state-of-the-art, with which they can research techniques to further improve effectiveness and reduce cost.
Paper Structure (43 sections, 7 equations, 4 figures, 19 tables)

This paper contains 43 sections, 7 equations, 4 figures, 19 tables.

Table of Contents

  1. Introduction
  2. Study Design
  3. Overview and Research Questions
  4. Studied Mutant Generation Approaches
  5. Traditional Approaches
  6. LLM-based Approaches
  7. Datasets
  8. Evaluation Metrics
  9. RQ1: Effectiveness
  10. RQ2: Validity
  11. RQ3: Efficiency
  12. Mutant Generation Setups
  13. Evaluation Results
  14. RQ1: Performance on Effectiveness
  15. RQ1.1 Detectability
  16. ...and 28 more sections

Figures (4)

  • Figure 1: Our New Prompt Template for Mutant Generation
  • Figure 2: Number of Newly Introduced AST Nodes of the Studied Approaches and the Few-Shot Examples
  • Figure 3: Distribution of the AST Node Types of Origin Code for Non-Compilable Mutants
  • Figure 4: The Modified Prompt Template for High-Order Mutant Generation