Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Laboratory to Real-World Applications: Benchmarking Agentic Code Reasoning at the Repository Level

Jia Li, Yuxin Su, Michael R. Lyu

TL;DR

This paper tackles the challenge of evaluating repository-level reasoning in autonomous code agents, addressing the gap between isolated code reasoning and real-world software ecosystems. It introduces RepoReason, a white-box diagnostic benchmark that uses abductive assertion verification and an execution-driven mutation framework to regenerate ground-truth states while preventing memorization. A cognitive diagnostic framework with three metrics—$ESV$ (reading load), $MCL$ (simulation depth), and $DFI$ (integration width)—quantifies reasoning bottlenecks and reveals an aggregation deficit, with $DFI$ being the principal limiting factor for frontier models. Evaluations across diverse models on real-world Python repositories demonstrate context overload and long-horizon state-tracking challenges, offering granular insights to guide the next generation of agentic software engineering tools.

Abstract

As large language models (LLMs) evolve into autonomous agents, evaluating repository-level reasoning, the ability to maintain logical consistency across massive, real-world, interdependent file systems, has become critical. Current benchmarks typically fluctuate between isolated code snippets and black-box evaluations. We present RepoReason, a white-box diagnostic benchmark centered on abductive assertion verification. To eliminate memorization while preserving authentic logical depth, we implement an execution-driven mutation framework that utilizes the environment as a semantic oracle to regenerate ground-truth states. Furthermore, we establish a fine-grained diagnostic system using dynamic program slicing, quantifying reasoning via three orthogonal metrics: $ESV$ (reading load), $MCL$ (simulation depth), and $DFI$ (integration width). Comprehensive evaluations of frontier models (e.g., Claude-4.5-Sonnet, DeepSeek-v3.1-Terminus) reveal a prevalent aggregation deficit, where integration width serves as the primary cognitive bottleneck. Our findings provide granular white-box insights for optimizing the next generation of agentic software engineering.

From Laboratory to Real-World Applications: Benchmarking Agentic Code Reasoning at the Repository Level

TL;DR

This paper tackles the challenge of evaluating repository-level reasoning in autonomous code agents, addressing the gap between isolated code reasoning and real-world software ecosystems. It introduces RepoReason, a white-box diagnostic benchmark that uses abductive assertion verification and an execution-driven mutation framework to regenerate ground-truth states while preventing memorization. A cognitive diagnostic framework with three metrics— (reading load), (simulation depth), and (integration width)—quantifies reasoning bottlenecks and reveals an aggregation deficit, with being the principal limiting factor for frontier models. Evaluations across diverse models on real-world Python repositories demonstrate context overload and long-horizon state-tracking challenges, offering granular insights to guide the next generation of agentic software engineering tools.

Abstract

As large language models (LLMs) evolve into autonomous agents, evaluating repository-level reasoning, the ability to maintain logical consistency across massive, real-world, interdependent file systems, has become critical. Current benchmarks typically fluctuate between isolated code snippets and black-box evaluations. We present RepoReason, a white-box diagnostic benchmark centered on abductive assertion verification. To eliminate memorization while preserving authentic logical depth, we implement an execution-driven mutation framework that utilizes the environment as a semantic oracle to regenerate ground-truth states. Furthermore, we establish a fine-grained diagnostic system using dynamic program slicing, quantifying reasoning via three orthogonal metrics: (reading load), (simulation depth), and (integration width). Comprehensive evaluations of frontier models (e.g., Claude-4.5-Sonnet, DeepSeek-v3.1-Terminus) reveal a prevalent aggregation deficit, where integration width serves as the primary cognitive bottleneck. Our findings provide granular white-box insights for optimizing the next generation of agentic software engineering.
Paper Structure (32 sections, 2 equations, 4 figures, 5 tables)

This paper contains 32 sections, 2 equations, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Code Generation
  4. Code Reasoning
  5. Agentic Software Engineering
  6. Summary Comparison
  7. RepoReason: Design and Construction
  8. Design Philosophy: Assertions as Semantic Anchors
  9. Foundation: Real-world Repository Curation
  10. Repository Selection
  11. File-Level Semantic Filtering
  12. Countering Memorization: The Execution-Driven Mutation Engine
  13. Countering Superficiality: Trace-Based Structural Filtering
  14. Countering Ambiguity: The Deterministic Value Protocol
  15. Cognitive Diagnostic Framework
  16. ...and 17 more sections

Figures (4)

  • Figure 1: The overall architecture and multi-stage pipeline of RepoReason, encompassing repository curation, structural filtering, execution-driven mutation, task instantiation, and diagnostic evaluation.
  • Figure 2: Performance trajectories of frontier LLM agents across three orthogonal cognitive metrics: (a) Effective Sliced Volume (ESV), (b) Mutation Chain Length (MCL), and (c) Dependency Fan-in (DFI).
  • Figure 3: Performance of LLM agents across different reasoning scopes.
  • Figure 4: Empirical distribution of the three cognitive metrics across the $\text{RepoReason}$ dataset: (a) $\text{ESV}$, (b) $\text{MCL}$, and (c) $\text{DFI}$.