Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LoCoBench-Agent: An Interactive Benchmark for LLM Agents in Long-Context Software Engineering

Jielin Qiu, Zuxin Liu, Zhiwei Liu, Rithesh Murthy, Jianguo Zhang, Haolin Chen, Shiyu Wang, Ming Zhu, Liangwei Yang, Juntao Tan, Roshan Ram, Akshara Prabhakar, Tulika Awalgaonkar, Zixiang Chen, Zhepeng Cen, Cheng Qian, Shelby Heinecke, Weiran Yao, Silvio Savarese, Caiming Xiong, Huan Wang

TL;DR

LoCoBench-Agent tackles the evaluation gap for autonomous LLM software engineering agents operating in long, interactive development contexts. It extends the static LoCoBench scenarios into rich multi-turn environments with long-context memory management, an 8-tool interactive framework, and a bias-free nine-metric evaluation pipeline spanning 10K–1M token contexts and up to 50 conversation turns. The framework reveals fundamental trade-offs between comprehension and efficiency, highlights the pivotal role of strategic tool usage and memory management, and identifies mature versus underdeveloped capabilities such as cross-file consistency and long-range dependency resolution. By offering large-scale, transparent, and reusable evaluation data, LoCoBench-Agent lays a rigorous foundation for advancing autonomous software development at scale and informs architectural directions for future agent systems.

Abstract

As large language models (LLMs) evolve into sophisticated autonomous agents capable of complex software development tasks, evaluating their real-world capabilities becomes critical. While existing benchmarks like LoCoBench~\cite{qiu2025locobench} assess long-context code understanding, they focus on single-turn evaluation and cannot capture the multi-turn interactive nature, tool usage patterns, and adaptive reasoning required by real-world coding agents. We introduce \textbf{LoCoBench-Agent}, a comprehensive evaluation framework specifically designed to assess LLM agents in realistic, long-context software engineering workflows. Our framework extends LoCoBench's 8,000 scenarios into interactive agent environments, enabling systematic evaluation of multi-turn conversations, tool usage efficiency, error recovery, and architectural consistency across extended development sessions. We also introduce an evaluation methodology with 9 metrics across comprehension and efficiency dimensions. Our framework provides agents with 8 specialized tools (file operations, search, code analysis) and evaluates them across context lengths ranging from 10K to 1M tokens, enabling precise assessment of long-context performance. Through systematic evaluation of state-of-the-art models, we reveal several key findings: (1) agents exhibit remarkable long-context robustness; (2) comprehension-efficiency trade-off exists with negative correlation, where thorough exploration increases comprehension but reduces efficiency; and (3) conversation efficiency varies dramatically across models, with strategic tool usage patterns differentiating high-performing agents. As the first long-context LLM agent benchmark for software engineering, LoCoBench-Agent establishes a rigorous foundation for measuring agent capabilities, identifying performance gaps, and advancing autonomous software development at scale.

LoCoBench-Agent: An Interactive Benchmark for LLM Agents in Long-Context Software Engineering

TL;DR

LoCoBench-Agent tackles the evaluation gap for autonomous LLM software engineering agents operating in long, interactive development contexts. It extends the static LoCoBench scenarios into rich multi-turn environments with long-context memory management, an 8-tool interactive framework, and a bias-free nine-metric evaluation pipeline spanning 10K–1M token contexts and up to 50 conversation turns. The framework reveals fundamental trade-offs between comprehension and efficiency, highlights the pivotal role of strategic tool usage and memory management, and identifies mature versus underdeveloped capabilities such as cross-file consistency and long-range dependency resolution. By offering large-scale, transparent, and reusable evaluation data, LoCoBench-Agent lays a rigorous foundation for advancing autonomous software development at scale and informs architectural directions for future agent systems.

Abstract

As large language models (LLMs) evolve into sophisticated autonomous agents capable of complex software development tasks, evaluating their real-world capabilities becomes critical. While existing benchmarks like LoCoBench~\cite{qiu2025locobench} assess long-context code understanding, they focus on single-turn evaluation and cannot capture the multi-turn interactive nature, tool usage patterns, and adaptive reasoning required by real-world coding agents. We introduce \textbf{LoCoBench-Agent}, a comprehensive evaluation framework specifically designed to assess LLM agents in realistic, long-context software engineering workflows. Our framework extends LoCoBench's 8,000 scenarios into interactive agent environments, enabling systematic evaluation of multi-turn conversations, tool usage efficiency, error recovery, and architectural consistency across extended development sessions. We also introduce an evaluation methodology with 9 metrics across comprehension and efficiency dimensions. Our framework provides agents with 8 specialized tools (file operations, search, code analysis) and evaluates them across context lengths ranging from 10K to 1M tokens, enabling precise assessment of long-context performance. Through systematic evaluation of state-of-the-art models, we reveal several key findings: (1) agents exhibit remarkable long-context robustness; (2) comprehension-efficiency trade-off exists with negative correlation, where thorough exploration increases comprehension but reduces efficiency; and (3) conversation efficiency varies dramatically across models, with strategic tool usage patterns differentiating high-performing agents. As the first long-context LLM agent benchmark for software engineering, LoCoBench-Agent establishes a rigorous foundation for measuring agent capabilities, identifying performance gaps, and advancing autonomous software development at scale.

Paper Structure

This paper contains 52 sections, 10 equations, 11 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Design of LoCoBench-Agent
  3. From LoCoBench to LoCoBench-Agent
  4. Long-Context Memory Management
  5. Interactive Agent Framework
  6. Benchmark Statistics
  7. Comparison with Existing Benchmarks
  8. Evaluation Pipeline
  9. Evaluation Metrics
  10. Comprehension Metrics
  11. Efficiency Metrics
  12. Rescaling and Aggregation
  13. Evaluation
  14. Models Evaluated
  15. Evaluation Protocol
  16. ...and 37 more sections

Figures (11)

  • Figure 1: LoCoBench-Agent Benchmark Statistics. Comprehensive statistics across 8,000 scenarios and 1,000 unique projects. (a) Project scale distributions demonstrate diversity ranging. (b) Language coverage ensures balanced evaluation across 10 programming languages with representative LOC and file count distributions.
  • Figure 2: Overall model performance comparison. Left: LCBA-Comprehension scores across six models. Right: LCBA-Efficiency scores showing the trade-off between comprehension and efficiency dimensions.
  • Figure 3: Top three models across nine evaluation metrics. Per-axis scaling reveals capability maturity across comprehension and efficiency dimensions.
  • Figure 4: Comprehension-efficiency trade-off scatter plot. Diagonal line represents the Pareto frontier. Negative correlation indicates fundamental architectural tension.
  • Figure 5: Long-context performance analysis across difficulty levels. Top-left: Comprehension vs context length. Top-right: Efficiency vs context length. Bottom-left: Per-difficulty comparison. Bottom-right: Degradation summary (Easy→Expert).
  • ...and 6 more figures