Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mutation-Guided LLM-based Test Generation at Meta

Christopher Foster, Abhishek Gulati, Mark Harman, Inna Harper, Ke Mao, Jillian Ritchey, Hervé Robert, Shubho Sengupta

TL;DR

The paper tackles targeted automated test generation by uniting mutation testing with LLM-driven test creation to harden code against specific concerns, notably privacy. ACH uses an agenetic workflow with a fault-creation LLM, an equivalence detector, and a test generator, producing buildable tests that aim to kill mutants and reduce regressions, demonstrated at industrial scale across 7 Meta platforms and 10,795 classes. Results show 9,095 mutants and 571 tests, with 73% engineer acceptance and 36% privacy relevance; equivalence-detection precision and recall are $0.79$ and $0.47$, rising to $0.95$ and $0.96$ with preprocessing, indicating practical viability. The work substantiates the value of mutation-guided, LLM-based test generation for safety-critical concerns and outlines open challenges in equivalence handling, coverage trade-offs, and oracle inference, suggesting a path toward broader applicability and performance improvements in industry-scale software testing.

Abstract

This paper describes Meta's ACH system for mutation-guided LLM-based test generation. ACH generates relatively few mutants (aka simulated faults), compared to traditional mutation testing. Instead, it focuses on generating currently undetected faults that are specific to an issue of concern. From these currently uncaught faults, ACH generates tests that can catch them, thereby `killing' the mutants and consequently hardening the platform against regressions. We use privacy concerns to illustrate our approach, but ACH can harden code against {\em any} type of regression. In total, ACH was applied to 10,795 Android Kotlin classes in 7 software platforms deployed by Meta, from which it generated 9,095 mutants and 571 privacy-hardening test cases. ACH also deploys an LLM-based equivalent mutant detection agent that achieves a precision of 0.79 and a recall of 0.47 (rising to 0.95 and 0.96 with simple pre-processing). ACH was used by Messenger and WhatsApp test-a-thons where engineers accepted 73% of its tests, judging 36% to privacy relevant. We conclude that ACH hardens code against specific concerns and that, even when its tests do not directly tackle the specific concern, engineers find them useful for their other benefits.

Mutation-Guided LLM-based Test Generation at Meta

TL;DR

The paper tackles targeted automated test generation by uniting mutation testing with LLM-driven test creation to harden code against specific concerns, notably privacy. ACH uses an agenetic workflow with a fault-creation LLM, an equivalence detector, and a test generator, producing buildable tests that aim to kill mutants and reduce regressions, demonstrated at industrial scale across 7 Meta platforms and 10,795 classes. Results show 9,095 mutants and 571 tests, with 73% engineer acceptance and 36% privacy relevance; equivalence-detection precision and recall are and , rising to and with preprocessing, indicating practical viability. The work substantiates the value of mutation-guided, LLM-based test generation for safety-critical concerns and outlines open challenges in equivalence handling, coverage trade-offs, and oracle inference, suggesting a path toward broader applicability and performance improvements in industry-scale software testing.

Abstract

This paper describes Meta's ACH system for mutation-guided LLM-based test generation. ACH generates relatively few mutants (aka simulated faults), compared to traditional mutation testing. Instead, it focuses on generating currently undetected faults that are specific to an issue of concern. From these currently uncaught faults, ACH generates tests that can catch them, thereby `killing' the mutants and consequently hardening the platform against regressions. We use privacy concerns to illustrate our approach, but ACH can harden code against {\em any} type of regression. In total, ACH was applied to 10,795 Android Kotlin classes in 7 software platforms deployed by Meta, from which it generated 9,095 mutants and 571 privacy-hardening test cases. ACH also deploys an LLM-based equivalent mutant detection agent that achieves a precision of 0.79 and a recall of 0.47 (rising to 0.95 and 0.96 with simple pre-processing). ACH was used by Messenger and WhatsApp test-a-thons where engineers accepted 73% of its tests, judging 36% to privacy relevant. We conclude that ACH hardens code against specific concerns and that, even when its tests do not directly tackle the specific concern, engineers find them useful for their other benefits.
Paper Structure (14 sections, 2 figures, 8 tables)

This paper contains 14 sections, 2 figures, 8 tables.

Table of Contents

  1. Introduction
  2. The ACH System
  3. Results from Meta's ACH Deployment
  4. Engineers' Evaluation of ACH
  5. Initial Trial
  6. Experience from Privacy Test-a-thons
  7. The Equivalent Mutant Problem
  8. Equivalent Mutants Have no Impact on ACH
  9. Detecting Equivalent Mutants
  10. Equivalence Detector Precision and Recall
  11. The Importance of Mutation Testing
  12. Related Work
  13. Open Problems
  14. Conclusions

Figures (2)

  • Figure 1: Top level architecture of the principal ACH components. The dotted section denotes a (slightly modified) version of the TestGen-LLM tool, on which we previously reported mhetal:TestGen-LLM. The workflow preceding this, shown above in the figure, is the additional agenetic workflow for generating candidate faults to drive the generation of tests. Solid rectangles denote components that are fully automated but entirely rule-based (and therefore do not use LLMs).
  • Figure 2: Likert scale instructions given to code reviewers to score test cases according to their privacy relevance.