Dynamic Cogeneration of Bug Reproduction Test in Agentic Program Repair

TL;DR

Bug Reproduction Tests (BRTs) are integrated into agentic Automated Program Repair (APR) as cogeneration—the simultaneous generation of a fix and a BRT within a single patch. The authors study Passerine-based cogeneration strategies (TDD, TLD, Freeform) across 120 Google bugs, showing cogeneration yields plausible fixes and BRTs at least as often as dedicated baselines, with Freeform performing best. They introduce test-aware patch selectors to prioritize patches containing both a plausible fix and BRT, and analyze root causes of cogeneration failures, offering mitigation strategies. The findings suggest cogeneration reduces engineering effort by unifying fix and BRT pipelines while maintaining repair quality, with practical implications for scalable AI-assisted debugging and patch validation.

Abstract

Bug Reproduction Tests (BRTs) have been used in many agentic Automated Program Repair (APR) systems, primarily for validating promising fixes and aiding fix generation. In practice, when developers submit a patch, they often implement the BRT alongside the fix. Our experience deploying agentic APR reveals that developers similarly desire a BRT within AI-generated patches to increase their confidence. However, canonical APR systems tend to generate BRTs and fixes separately, or focus on producing only the fix in the final patch. In this paper, we study agentic APR in the context of cogeneration, where the APR agent is instructed to generate both a fix and a BRT in the same patch. We evaluate the effectiveness of different cogeneration strategies on 120 human-reported bugs at Google and characterize different cogeneration strategies by their influence on APR agent behavior. We develop and evaluate patch selectors that account for test change information to select patches with plausible fixes (and plausible BRTs). Finally, we analyze the root causes of failed cogeneration trajectories. Importantly, we show that cogeneration allows the APR agent to generate BRTs for at least as many bugs as a dedicated BRT agent, without compromising the generation rate of plausible fixes, thereby reducing engineering effort in maintaining and coordinating separate generation pipelines for fix and BRT at scale.

Figures (11)

• Figure 1: Overview of our agentic APR system under cogeneration. The APR agent utilizes one of three cogeneration strategies: (1) Test-Driven Development (TDD), (2) Test-Last Development (TLD), or (3) Freeform. This process cogenerates patches containing both a code fix and a Bug Reproduction Test (BRT). These patches undergo a multi-stage validation process by a suite of reviewers (spec-based, smell, BRT, and build & test) to assess validity and reproducibility. Finally, a patch selection strategy is used to select the most promising patch among the candidates.
• Figure 2: Additional system instruction for cogeneration.
• Figure 3: Additional TDD instruction.
• Figure 4: Additional TLD instruction.
• Figure 5: Effectiveness measured by success@k metrics (§\ref{['sec:setup:rq:rq1']}). All cogeneration strategies outperform instructing the agent to generate a fix alone or BRT alone. Freeform cogeneration achieves the highest (pass & plausible BRT)@20 and pass@20.