Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CR-Bench: Evaluating the Real-World Utility of AI Code Review Agents

Kristen Pereira, Neelabh Sinha, Rajat Ghosh, Debojyoti Dutta

TL;DR

It is found that code review agents can exhibit a low signal-to-noise ratio when designed to identify all hidden issues, obscuring true progress and developer productivity when measured solely by resolution rates.

Abstract

Recent advances in frontier large language models have enabled code review agents that operate in open-ended, reasoning-intensive settings. However, the lack of standardized benchmarks and granular evaluation protocols makes it difficult to assess behavior of code review agents beyond coarse success metrics, particularly for tasks where false positives are costly. To address this gap, we introduce CR-Bench, a benchmarking dataset, and CR-Evaluator, a fine-grained evaluation pipeline for code review agents. Using these tools, we conduct a preliminary study evaluating both a single-shot agent and a Reflexion-based agent across two frontier models. We find that code review agents can exhibit a low signal-to-noise ratio when designed to identify all hidden issues, obscuring true progress and developer productivity when measured solely by resolution rates. Our analysis identifies the hidden trade-off between issue resolution and spurious findings, revealing a frontier that constrains effective agent design. Together, CR-Bench and CR-Evaluator provide a timely foundation for studying and developing code review agents as LLM-based systems transition from controlled benchmarks to real-world software engineering workflows.

CR-Bench: Evaluating the Real-World Utility of AI Code Review Agents

TL;DR

It is found that code review agents can exhibit a low signal-to-noise ratio when designed to identify all hidden issues, obscuring true progress and developer productivity when measured solely by resolution rates.

Abstract

Recent advances in frontier large language models have enabled code review agents that operate in open-ended, reasoning-intensive settings. However, the lack of standardized benchmarks and granular evaluation protocols makes it difficult to assess behavior of code review agents beyond coarse success metrics, particularly for tasks where false positives are costly. To address this gap, we introduce CR-Bench, a benchmarking dataset, and CR-Evaluator, a fine-grained evaluation pipeline for code review agents. Using these tools, we conduct a preliminary study evaluating both a single-shot agent and a Reflexion-based agent across two frontier models. We find that code review agents can exhibit a low signal-to-noise ratio when designed to identify all hidden issues, obscuring true progress and developer productivity when measured solely by resolution rates. Our analysis identifies the hidden trade-off between issue resolution and spurious findings, revealing a frontier that constrains effective agent design. Together, CR-Bench and CR-Evaluator provide a timely foundation for studying and developing code review agents as LLM-based systems transition from controlled benchmarks to real-world software engineering workflows.
Paper Structure (17 sections, 6 equations, 4 figures, 4 tables, 1 algorithm)

This paper contains 17 sections, 6 equations, 4 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Formalism
  4. CR-Bench: Benchmarking Dataset for Code Review
  5. Transformation of SWE-Bench to CR-Bench
  6. Taxonomy
  7. CR-Evaluator
  8. Experiments
  9. Results
  10. CR-Bench Statistics
  11. Performance Analysis
  12. Recall Distribution Across Taxonomy Components
  13. Conclusion
  14. Future Work
  15. Appendix: CR-Bench Prompts
  16. ...and 2 more sections

Figures (4)

  • Figure 1: Example of an instance from CR-Bench dataset: In a pull request, a comment should be made to address the specified bug. The fix is removing the lines of code marked in RED and adding the ones marked in GREEN.
  • Figure 2: CR-Bench is a dataset for transforming real-world software defects into an objective code review benchmark with full Pull Request (PR) context and a multi-dimensional taxonomy of Category, Impact, Severity. CR-Evaluator is a method of evaluating code review agents and measuring them in terms of both performance and developer acceptance.
  • Figure 3: Comparative analysis of bug category, severity, and impact distributions between the verified subset and the full CR-Bench corpus (Note: RFF = Requirements, Features and Functionality, IIS = Interface, Integration and System).
  • Figure 4: PR Bug Recall analysis across Category, Impact, and Severity for CR-Bench-verified (Note: RFF = Requirements, Features and Functionality, IIS = Interface, Integration and System).