Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SWE-PRBench: Benchmarking AI Code Review Quality Against Pull Request Feedback

Deepak Kumar

Abstract

We introduce SWE-PRBench, a benchmark of 350 pull requests with human-annotated ground truth for evaluating AI code review quality. Evaluated against an LLM-as-judge framework validated at kappa=0.75, 8 frontier models detect only 15-31% of human-flagged issues on the diff-only configuration, demonstrating that AI code review remains far below human expert performance despite strong results on code generation benchmarks. Pull requests are drawn from active open-source repositories, filtered from 700 candidates using a Repository Quality Score, and evaluated under three frozen context configurations: diff only (config_A), diff with file content (config_B), and full context (config_C), enabling systematic ablation of context provision strategies. All 8 models degrade monotonically from config_A to config_C, even when context is provided via structured semantic layers including AST-extracted function context and import graph resolution. The dominant mechanism is a collapse of Type2_Contextual issue detection at config_B, consistent with attention dilution in long contexts: a structured 2,000-token diff-with-summary prompt outperforms a 2,500-token full-context prompt enriched with execution context, behaviour mapping, and test signatures across all 8 models. The top four models are statistically indistinguishable (mean score 0.147-0.153) while a clear tier gap separates them from the remaining four (mean score <= 0.113). Dataset, contexts, annotations, and evaluation harness are released publicly.

SWE-PRBench: Benchmarking AI Code Review Quality Against Pull Request Feedback

Abstract

We introduce SWE-PRBench, a benchmark of 350 pull requests with human-annotated ground truth for evaluating AI code review quality. Evaluated against an LLM-as-judge framework validated at kappa=0.75, 8 frontier models detect only 15-31% of human-flagged issues on the diff-only configuration, demonstrating that AI code review remains far below human expert performance despite strong results on code generation benchmarks. Pull requests are drawn from active open-source repositories, filtered from 700 candidates using a Repository Quality Score, and evaluated under three frozen context configurations: diff only (config_A), diff with file content (config_B), and full context (config_C), enabling systematic ablation of context provision strategies. All 8 models degrade monotonically from config_A to config_C, even when context is provided via structured semantic layers including AST-extracted function context and import graph resolution. The dominant mechanism is a collapse of Type2_Contextual issue detection at config_B, consistent with attention dilution in long contexts: a structured 2,000-token diff-with-summary prompt outperforms a 2,500-token full-context prompt enriched with execution context, behaviour mapping, and test signatures across all 8 models. The top four models are statistically indistinguishable (mean score 0.147-0.153) while a clear tier gap separates them from the remaining four (mean score <= 0.113). Dataset, contexts, annotations, and evaluation harness are released publicly.

Paper Structure

This paper contains 35 sections, 2 equations, 4 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Code Generation and Repair Benchmarks
  4. Code Review Automation
  5. LLM-as-Judge Methodology
  6. The SWE-PRBench Dataset
  7. Repository Selection and Quality Scoring
  8. PR Collection and Filtering
  9. PR Review Value Score
  10. Difficulty Taxonomy
  11. Contamination Mitigation
  12. Dataset Statistics
  13. Context Configurations
  14. The V1 Problem
  15. Three Fixed Configurations
  16. ...and 20 more sections

Figures (4)

  • Figure 1: SWE-PRBench leaderboard with 95% bootstrap CIs. Top four models cluster ($\bar{s}$ 0.147--0.153); clear tier gap at rank 4/5.
  • Figure 2: Composite score by model and context configuration. All 8 models degrade A$\to$C; the A$\to$B drop is the largest step.
  • Figure 3: Difficulty $\times$ config heatmap for three models. The Type2 collapse at config_B is visible across all three.
  • Figure 4: Mean hallucination rate (FPR) across config_A/B/C. GPT-4o is most precise (0.193); Llama fabricates at 0.417.