ContextBench: A Benchmark for Context Retrieval in Coding Agents

TL;DR

ContextBench addresses the gap in evaluating LLM-based coding agents by focusing on context retrieval rather than solely on end-to-end patch success. It introduces a large, human-annotated benchmark of 1,136 issue-resolution tasks across 66 repositories and 8 languages, plus an automated pipeline that aligns agent-retrieved contexts with gold contexts at file, block, and line granularity. Evaluations of four frontier LLMs and five coding agents reveal that sophisticated scaffolding often yields only marginal retrieval gains, while LLMs tend to maximize recall at the expense of precision, creating noise and gaps between retrieved and utilized context. The benchmark provides intermediate signals through trajectory-aware metrics, enabling more interpretable and reliable improvements in software-engineering AI systems.

Abstract

LLM-based coding agents have shown strong performance on automated issue resolution benchmarks, yet existing evaluations largely focus on final task success, providing limited insight into how agents retrieve and use code context during problem solving. We introduce ContextBench, a process-oriented evaluation of context retrieval in coding agents. ContextBench consists of 1,136 issue-resolution tasks from 66 repositories across eight programming languages, each augmented with human-annotated gold contexts. We further implement an automated evaluation framework that tracks agent trajectories and measures context recall, precision, and efficiency throughout issue resolution. Using ContextBench, we evaluate four frontier LLMs and five coding agents. Our results show that sophisticated agent scaffolding yields only marginal gains in context retrieval ("The Bitter Lesson" of coding agents), LLMs consistently favor recall over precision, and substantial gaps exist between explored and utilized context. ContextBench augments existing end-to-end benchmarks with intermediate gold-context metrics that unbox the issue-resolution process. These contexts offer valuable intermediate signals for guiding LLM reasoning in software tasks. Data and code are available at: https://cioutn.github.io/context-bench/.

Figures (12)

• Figure 1: Radar plots comparing context retrieval performance on ContextBench. (a) Coding agent comparison and (b) LLM comparison, both evaluated at file-, block-, and line-level granularity in terms of precision, recall, and F1. Across both LLMs and agents, higher recall is consistently favored over precision.
• Figure 2: An overview of the ContextBench construction pipeline.ContextBench is curated through three key steps: (1) Task Deduplication removes exact and near-duplicate tasks from multiple issue resolution benchmarks using rule-based and embedding-based detection. (2) Task Selection identifies challenging tasks based on agent solvability and the scope and dispersion of edits in ground-truth patches. (3) Expert Annotation employs a human-in-the-loop procedure in which expert developers trace code dependencies to construct gold contexts, which are validated through LLM-based patch generation and inter-annotator agreement checks.
• Figure 3: An illustrative example of the human-verified gold context. This example is derived from the issue task associated with Pull Request #4009 in the langchain-ai/langchain repository. The blue, red, and green arrows represent the manually annotated contextual dependencies associated with variable, function, and class invocations, respectively.
• Figure 4: The data flow during the ContextBench construction pipeline. We construct ContextBench step by step, starting from task deduplication and selection to gold context annotation.
• Figure 6: : Taxonomy of failure types