Abstain and Validate: A Dual-LLM Policy for Reducing Noise in Agentic Program Repair

TL;DR

The paper addresses noise in agentic Automated Program Repair (APR) by introducing two complementary LLM-based policies: bug abstention to avoid attempting repairs on unlikely bugs, and patch validation to filter generated patches before human review. Through experiments on Google's bug sets, the authors demonstrate that these policies, alone or in combination, meaningfully increase the likelihood that patches shown to developers are correct, enabling more reliable industrial deployment. The work provides formalizations, data-driven insights, and practical guidance for configuring a repair funnel that balances throughput and patch quality. Overall, the dual-policy framework offers a scalable path to reducing wasted reviewer effort and improving trust in automated code changes.

Abstract

Agentic Automated Program Repair (APR) is increasingly tackling complex, repository-level bugs in industry, but ultimately agent-generated patches still need to be reviewed by a human before committing them to ensure they address the bug. Showing unlikely patches to developers can lead to substantial noise, wasting valuable developer time and eroding trust in automated code changes. We introduce two complementary LLM-based policies to reduce such noise: bug abstention and patch validation policies. Bug abstention excludes bugs that the agentic APR system is unlikely to fix. Patch validation rejects patches that are unlikely to be a good fix for the given bug. We evaluate both policies on three sets of bugs from Google's codebase, and their candidate patches generated by an internal agentic APR system. On a set of 174 human-reported bugs, removing bugs and patch trajectories rejected by our policies can raise success rates by up to 13 percentage points and 15 percentage points, respectively, and by up to 39 percentage points in combination. On null pointer exceptions and sanitizer-reported bugs with machine-generated bug reports, patch validation also improves average single-sample success rates. This two-policy approach provides a practical path to the reliable, industrial-scale deployment of agentic APR systems.

Figures (18)

• Figure 1: Bug abstention rejects issues that are unlikely to be fixed by our APR system. After the agent generates patches, validation decides whether these are likely to have addressed the issue. Jointly they reduce developer noise from APR.
• Figure 2: Using abstention with guidelines better aligns model probabilities with observed (left-skewed) rates.
• Figure 3: Both abstention variants outperform randomly sampling to identify bugs with at least one successful repair. Adding guidelines helps. (AP=average precision)
• Figure 4: Abstention can raise fail-to-pass@k by removing bugs that are less likely to succeed with our APR system.
• Figure 5: Patch correctness score allows more aggressive validation, with precision up to 3$\times$ the success prevalence.