Agentic Rubrics as Contextual Verifiers for SWE Agents

TL;DR

Verification signals for SWE agents are central to training and efficient inference but are hard to scale when relying on code execution. The authors propose Agentic Rubrics, which generate a repository-grounded rubric with four axes and score patches execution-free, enabling parallel test-time scaling. On SWE-Bench Verified, Agentic Rubrics outperform strong baselines and align well with ground-truth tests while exposing issues tests may miss; ablations show that repository-grounding and rubric-agent quality are essential, with open-weight models capable of distilling the approach. Overall, the method offers a scalable, interpretable, and granular verification signal that can improve post-training rewards and patch selection in SWE agents.

Abstract

Verification is critical for improving agents: it provides the reward signal for Reinforcement Learning and enables inference-time gains through Test-Time Scaling (TTS). Despite its importance, verification in software engineering (SWE) agent settings often relies on code execution, which can be difficult to scale due to environment setup overhead. Scalable alternatives such as patch classifiers and heuristic methods exist, but they are less grounded in codebase context and harder to interpret. To this end, we explore Agentic Rubrics: an expert agent interacts with the repository to create a context-grounded rubric checklist, and candidate patches are then scored against it without requiring test execution. On SWE-Bench Verified under parallel TTS evaluation, Agentic Rubrics achieve a score of 54.2% on Qwen3-Coder-30B-A3B and 40.6% on Qwen3-32B, with at least a +3.5 percentage-point gain over the strongest baseline in our comparison set. We further analyze rubric behavior, showing that rubric scores are consistent with ground-truth tests while also flagging issues that tests do not capture. Our ablations show that agentic context gathering is essential for producing codebase-specific, unambiguous criteria. Together, these results suggest that Agentic Rubrics provide an efficient, scalable, and granular verification signal for SWE agents.

Figures (11)

• Figure 1: Agentic rubric pipeline. In the rubric-generation phase (left), a rubric agent inspects the codebase and PR description using repository tools, then produces a rubric.yaml organized along four rubric axes (File Change, Spec Alignment, Integrity, Runtime). In the verification phase (right), a SWE agent proposes a patch, which is graded against the rubric to yield an execution-free verifier score.
• Figure 2: (Left) Best@16 resolution (%) with $K=16$ rollouts for Qwen3-32B and Qwen3-Coder-30B-A3B. Verifier signals are generated with Claude Sonnet-4.5; LLM judging uses GPT-5 (low reasoning). (Right) Best@K scaling curves for Qwen3-32B rollouts under different verifiers, with numbers averaged over 100 trials.
• Figure 3: Distribution of Weighted Rubric score for Qwen3-32B rollouts on Sonnet-4.5 generated agentic rubrics, for both correct (Ground Truth Tests Pass - Green) and incorrect (Ground-Truth tests Fail - Red). Rubric scores are well aligned with the GT Test correctness signal, awarding lower score for incorrect patches and higher score for correct ones, while providing a denser score distribution.
• Figure 4: Category-wise distribution of Sonnet-4.5 rubric scores on Qwen3-32B rollouts. Incorrect patches (GT Test Failed, in red) score lower on File Change (Edit scope) and Spec Alignment (Satisfying prompt requirements) and Runtime issues, but still good preserving codebase integrity and avoid cheating. Patches that pass ground-truth tests (GT Test Passed, in green) have a very high spec-alignment and integrity score but still suffer from edit scope and in some cases, have issues in runtime checks.
• Figure 5: Qualitative breakdown of agentic rubric utility relative to SWE-Bench Verified ground-truth tests. (a) In high-alignment cases, 78% of rubrics are high-utility (core semantics, API/compatibility, structure, edge coverage), with 22% low-utility (low-signal, over-specified, spec- or test-mismatched). (b) When tests pass but rubric scores are low, 54% of rubric failures are high-utility—often flagging missed root causes or missing edge-case coverage—while 46% reflect low-utility modes (over-specification, redundancy, rule mismatches, spec conflicts).