Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Agentic Reinforcement Learning for Real-World Code Repair

Siyu Zhu, Anastasiya Karpovich, Albert Chen, Jessica Koscheka, Shailesh Jannu, Di Wen, Yuqing Zhu, Rohit Jain, Alborz Geramifard

TL;DR

The paper tackles real-world code repair by building a verifiable, reproducible training and evaluation pipeline for heterogeneous repositories. It demonstrates that Qwen3-32B can be SFT-fine-tuned to approach GPT-4.1 performance in a full pipeline, and that RL on a simplified, matched environment yields 7–20 percentage-point gains, indicating potential for expert-like autonomous repair. However, cross-domain generalization remains a major challenge, and longer RL training can invite reward-exploitation behaviors, underscoring the need for robust evaluation and reward design. Overall, the work moves toward reliable AI-assisted code repair in real production settings by acknowledging environment mismatch and emphasizing execution-grounded learning with realistic data and infrastructure constraints.

Abstract

We tackle the challenge of training reliable code-fixing agents in real repositories, where complex builds and shifting dependencies make evaluation unstable. We developed a verifiable pipeline with success defined as post-fix build validation and improved reproducibility across ~1K real issues by pinning dependencies and disabling automatic upgrades. Building on this, we introduced a scalable simplified pipeline for large-scale reinforcement learning (RL). Using this setup, we supervised fine-tuned Qwen3-32B in the full pipeline and applied RL on top of the SFT model in the simplified environment. The SFT model distilled from GPT-4.1 trajectories performs on par while being 56x smaller, and RL added 7-20% absolute gains under matched train-test conditions. "Thinking mode" was on par or worse in our experiments. Both SFT and RL models failed to generalize across environments, highlighting the importance of matching train-test environments for building reliable real-world code-fixing agents.

Agentic Reinforcement Learning for Real-World Code Repair

TL;DR

The paper tackles real-world code repair by building a verifiable, reproducible training and evaluation pipeline for heterogeneous repositories. It demonstrates that Qwen3-32B can be SFT-fine-tuned to approach GPT-4.1 performance in a full pipeline, and that RL on a simplified, matched environment yields 7–20 percentage-point gains, indicating potential for expert-like autonomous repair. However, cross-domain generalization remains a major challenge, and longer RL training can invite reward-exploitation behaviors, underscoring the need for robust evaluation and reward design. Overall, the work moves toward reliable AI-assisted code repair in real production settings by acknowledging environment mismatch and emphasizing execution-grounded learning with realistic data and infrastructure constraints.

Abstract

We tackle the challenge of training reliable code-fixing agents in real repositories, where complex builds and shifting dependencies make evaluation unstable. We developed a verifiable pipeline with success defined as post-fix build validation and improved reproducibility across ~1K real issues by pinning dependencies and disabling automatic upgrades. Building on this, we introduced a scalable simplified pipeline for large-scale reinforcement learning (RL). Using this setup, we supervised fine-tuned Qwen3-32B in the full pipeline and applied RL on top of the SFT model in the simplified environment. The SFT model distilled from GPT-4.1 trajectories performs on par while being 56x smaller, and RL added 7-20% absolute gains under matched train-test conditions. "Thinking mode" was on par or worse in our experiments. Both SFT and RL models failed to generalize across environments, highlighting the importance of matching train-test environments for building reliable real-world code-fixing agents.
Paper Structure (23 sections, 10 figures, 1 table)

This paper contains 23 sections, 10 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLMs for Automated Code Repair.
  4. Agentic RL with Execution Feedback.
  5. Enterprise and Real-World Evaluation.
  6. Problem Formulation
  7. MDP Formulation
  8. Full Pipeline
  9. Simplified Pipeline
  10. Data
  11. Experimental Setup
  12. Experimental Results
  13. Policy Shift Analysis
  14. Conclusion
  15. Acknowledgments
  16. ...and 8 more sections

Figures (10)

  • Figure 1: Automated code-fixing pipeline. Each PR passes through build and validation; on failure, the system analyzes logs, retrieves candidate patches, selects a promising fix, and synthesizes a solution before retrying the build.
  • Figure 2: PR success rate with 95% confidence interval of various models on the test set for the full pipeline. The * in the name means execution with thinking.
  • Figure 3: PR success rate with 95% confidence interval of various models on the test set for the simplified VerL pipeline. The * in the name means execution with thinking.
  • Figure 4: Policy shift between Qwen3-32B base (left) and SFT+RL-fine-tuned (right) using a Markov chain transition probability matrix. The base model issued diverse tool calls with low task focus, while the fine-tuned model converged to focused sequences dominated by dependency_upgrade and validate_and_build, improving build success.
  • Figure 5: Token distribution across success, failure, and total runs with assistant thinking included.
  • ...and 5 more figures