FlakyGuard: Automatically Fixing Flaky Tests at Industry Scale

TL;DR

FlakyGuard addresses the context problem in LLM-based flaky-test repair by using LLM-guided exploration of dynamic call graphs to retrieve concise, relevant production-context for fixes. It constructs dynamic call graphs from runtime traces, applies a selective graph-search strategy to gather only relevant nodes, and combines this with AST-based test simplification and patch transplantation to generate robust patches. In large-scale industrial evaluation at Uber, FlakyGuard fixed 47.6% of reproducible flaky tests with 51.8% of patches accepted, outperforming state-of-the-art by at least 22%. Developers reported high usefulness of root-cause explanations, and the system has been deployed as an autonomous repair pipeline, illustrating practical viability for scalable flaky-test repair in industry.

Abstract

Flaky tests that non-deterministically pass or fail waste developer time and slow release cycles. While large language models (LLMs) show promise for automatically repairing flaky tests, existing approaches like FlakyDoctor fail in industrial settings due to the context problem: providing either too little context (missing critical production code) or too much context (overwhelming the LLM with irrelevant information). We present FlakyGuard, which addresses this problem by treating code as a graph structure and using selective graph exploration to find only the most relevant context. Evaluation on real-world flaky tests from industrial repositories shows that FlakyGuard repairs 47.6 % of reproducible flaky tests with 51.8 % of the fixes accepted by developers. Besides it outperforms state-of-the-art approaches by at least 22 % in repair success rate. Developer surveys confirm that 100 % find FlakyGuard's root cause explanations useful.

Figures (3)

• Figure 1: Overview of the FlakyGuard Workflow.
• Figure 2: Venn diagram for the fixes generated by different approaches
• Figure 3: Venn diagram for the fixes generated with different models