FeatureBench: Benchmarking Agentic Coding for Complex Feature Development

TL;DR

FeatureBench addresses the need for realistic, end-to-end evaluation of agentic coding by introducing an execution-based benchmark that focuses on feature development rather than bug fixing. It couples a test-driven task extraction pipeline with automatic environment generation to derive feature-oriented tasks from real Python repositories, producing 200 tasks and 3825 verifiable environments from 24 repositories. The paper benchmark compares multiple state-of-the-art models, revealing that even strong agents solve only a small fraction of tasks, which underscores the complexity of real-world feature development and long-horizon planning. The contributions include the benchmark itself, a scalable collection toolkit, and an in-depth analysis of challenges and directions for improving agentic coding capabilities in practice.

Abstract

Agents powered by large language models (LLMs) are increasingly adopted in the software industry, contributing code as collaborators or even autonomous developers. As their presence grows, it becomes important to assess the current boundaries of their coding abilities. Existing agentic coding benchmarks, however, cover a limited task scope, e.g., bug fixing within a single pull request (PR), and often rely on non-executable evaluations or lack an automated approach for continually updating the evaluation coverage. To address such issues, we propose FeatureBench, a benchmark designed to evaluate agentic coding performance in end-to-end, feature-oriented software development. FeatureBench incorporates an execution-based evaluation protocol and a scalable test-driven method that automatically derives tasks from code repositories with minimal human effort. By tracing from unit tests along a dependency graph, our approach can identify feature-level coding tasks spanning multiple commits and PRs scattered across the development timeline, while ensuring the proper functioning of other features after the separation. Using this framework, we curated 200 challenging evaluation tasks and 3825 executable environments from 24 open-source repositories in the first version of our benchmark. Empirical evaluation reveals that the state-of-the-art agentic model, such as Claude 4.5 Opus, which achieves a 74.4% resolved rate on SWE-bench, succeeds on only 11.0% of tasks, opening new opportunities for advancing agentic coding. Moreover, benefiting from our automated task collection toolkit, FeatureBench can be easily scaled and updated over time to mitigate data leakage. The inherent verifiability of constructed environments also makes our method potentially valuable for agent training.

Figures (23)

• Figure 1: a) The agent must implement a directly callable feature based on the task description and interface definitions, either by developing from scratch or extending an existing repository. b) Our benchmark shows that even Claude Opus 4.5 achieves only a 11.0% solution rate.
• Figure 2: Given a GitHub repository, our automated toolkit initializes the development environment via Docker. For each benchmark instance, it validates and selects fail-to-pass and pass-to-pass tests. Then, the system performs dynamic tracing to capture runtime behavior and construct an object dependency graph. Leveraging this graph, the toolkit synthesizes code patches, derives corresponding pre-solved codebases, and formulates final problem statements. This pipeline has yielded 200 benchmark tasks and 3825 executable environments from 24 GitHub repositories.
• Figure 3: Average numbers characterizing different attributes of a SWE-bench task instance, as well as our FeatureBench ($L_1$ set).
• Figure 4: Failure modes of the Claude Opus 4.5. Models marked with $^{\dagger}$ use low reasoning, and $^{\ddagger}$ use medium reasoning.
• Figure 5: An ablation study to evaluate the necessity of manual verification for the examples generated by our system. Models marked with $^{\dagger}$ use low reasoning, and $^{\ddagger}$ use medium reasoning.