Scalable Supervising Software Agents with Patch Reasoner

TL;DR

R4P reframes patch verification in SWE as a scalable, reasoning task that uses a group-wise RL objective to compare multiple patches and provide dense rewards, reducing dependence on heavy test environments. It achieves strong patch-verification accuracy (72.2%) and outperforms strong baselines, and it enables training a lightweight, test-free agent (Mini-SE) that attains meaningful improvements in Pass@1 (26.2%) and best-at metrics (32.8%) with test-time scaling. The approach dramatically speeds up verification (≈1s per patch) and broadens data utilization from unlabeled sources, supporting continual learning in real-world OSS. Together, R4P and Mini-SE demonstrate a practical, scalable alternative to test-based supervision for SWE agents, with clear guidance on deployment and limitations.

Abstract

While large language model agents have advanced software engineering tasks, the unscalable nature of existing test-based supervision is limiting the potential improvement of data scaling. The reason is twofold: (1) building and running test sandbox is rather heavy and fragile, and (2) data with high-coverage tests is naturally rare and threatened by test hacking via edge cases. In this paper, we propose R4P, a patch verifier model to provide scalable rewards for training and testing SWE agents via reasoning. We consider that patch verification is fundamentally a reasoning task, mirroring how human repository maintainers review patches without writing and running new reproduction tests. To obtain sufficient reference and reduce the risk of reward hacking, R4P uses a group-wise objective for RL training, enabling it to verify multiple patches against each other's modification and gain a dense reward for stable training. R4P achieves 72.2% Acc. for verifying patches from SWE-bench-verified, surpassing OpenAI o3. To demonstrate R4P's practicality, we design and train a lite scaffold, Mini-SE, with pure reinforcement learning where all rewards are derived from R4P. As a result, Mini-SE achieves 26.2% Pass@1 on SWE-bench-verified, showing a 10.0% improvement over the original Qwen3-32B. This can be further improved to 32.8% with R4P for test-time scaling. Furthermore, R4P verifies patches within a second, 50x faster than testing on average. The stable scaling curves of rewards and accuracy along with high efficiency reflect R4P's practicality.

Figures (14)

• Figure 1: In this paper, we explore a reasoning-based patch verification strategy to provide scalable supervision for software engineering agents. This approach (1) mitigates data scarcity caused by test quality requirements in open-source codebases, (2) removes the need for environment setup and makes data expansion costless, and (3) greatly reduces computational overhead compared to heavy test execution. We aim to leverage such imperfect yet easily scalable supervision to enhance model capability even after high-quality test data is exhausted.
• Figure 1: Comparison with general models. ("$*$" mark means point-wise patch verification)
• Figure 2: Mini-SE adopts an test-free issue-resolution-oriented tool design strategy: Search: input an entity name, output a file path and code snippet. Edit: input a file path, old code and new code snippet, output a diff patch. This design prevents the inefficient, redundant iterations caused by free-style exploration during training rollouts. It also removes self-testing and fixing when generating patches since R4P could take this role for penalizing wrong ones during training and selecting the correct one during inference. This further improves efficiency and avoid relying on sandbox.
• Figure 3: Comparision with specialized models. ("$*$" mark means pair-wise patch selection)
• Figure 4: Left: Mini-SE's training and testing rewards during RL training. Upper Right: Mini-SE's resolution rates w.r.t. different TTS strategy. Lower Right: R4P's performance on selecting DeepSWE's patches. The scaling curves illustrates R4P's practicality and generalizability.