Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ProjDevBench: Benchmarking AI Coding Agents on End-to-End Project Development

Pengrui Lu, Shiqi Zhang, Yunzhong Hou, Lyumanshan Ye, Chaoyi Huang, Zixi Chen, Ji Zeng, Hantao Jiang, Pengfei Liu, Yiwei Wang, Ming-Hsuan Yang

TL;DR

ProjDevBench presents an end-to-end benchmark for AI coding agents that builds full software repositories from high-level specifications. It employs a dual evaluation pipeline—execution on an Online Judge and LLM-assisted code review—to measure functional correctness and specification compliance across 20 tasks in 8 categories, using six agents and multiple LLM backends. The study reveals sizable gaps in end-to-end capabilities, including specification alignment, edge-case handling, and resource management, with an overall acceptance rate of 27.38% and a notable reliance on extended interaction that often hinders progress. By highlighting systematic failure modes and validating a scalable code-review process, ProjDevBench provides a practical framework to push autonomous software engineering toward reliable, specification-driven, multi-file project construction.

Abstract

Recent coding agents can generate complete codebases from simple prompts, yet existing evaluations focus on issue-level bug fixing and lag behind end-to-end development. We introduce ProjDevBench, an end-to-end benchmark that provides project requirements to coding agents and evaluates the resulting repositories. Combining Online Judge (OJ) testing with LLM-assisted code review, the benchmark evaluates agents on (1) system architecture design, (2) functional correctness, and (3) iterative solution refinement. We curate 20 programming problems across 8 categories, covering both concept-oriented tasks and real-world application scenarios, and evaluate six coding agents built on different LLM backends. Our evaluation reports an overall acceptance rate of 27.38%: agents handle basic functionality and data structures but struggle with complex system design, time complexity optimization, and resource management. Our benchmark is available at https://github.com/zsworld6/projdevbench.

ProjDevBench: Benchmarking AI Coding Agents on End-to-End Project Development

TL;DR

ProjDevBench presents an end-to-end benchmark for AI coding agents that builds full software repositories from high-level specifications. It employs a dual evaluation pipeline—execution on an Online Judge and LLM-assisted code review—to measure functional correctness and specification compliance across 20 tasks in 8 categories, using six agents and multiple LLM backends. The study reveals sizable gaps in end-to-end capabilities, including specification alignment, edge-case handling, and resource management, with an overall acceptance rate of 27.38% and a notable reliance on extended interaction that often hinders progress. By highlighting systematic failure modes and validating a scalable code-review process, ProjDevBench provides a practical framework to push autonomous software engineering toward reliable, specification-driven, multi-file project construction.

Abstract

Recent coding agents can generate complete codebases from simple prompts, yet existing evaluations focus on issue-level bug fixing and lag behind end-to-end development. We introduce ProjDevBench, an end-to-end benchmark that provides project requirements to coding agents and evaluates the resulting repositories. Combining Online Judge (OJ) testing with LLM-assisted code review, the benchmark evaluates agents on (1) system architecture design, (2) functional correctness, and (3) iterative solution refinement. We curate 20 programming problems across 8 categories, covering both concept-oriented tasks and real-world application scenarios, and evaluate six coding agents built on different LLM backends. Our evaluation reports an overall acceptance rate of 27.38%: agents handle basic functionality and data structures but struggle with complex system design, time complexity optimization, and resource management. Our benchmark is available at https://github.com/zsworld6/projdevbench.
Paper Structure (59 sections, 4 equations, 5 figures, 9 tables)

This paper contains 59 sections, 4 equations, 5 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. AI Coding Agents.
  4. Evolution of AI-Assisted SE Tasks.
  5. Code Generation Benchmarks.
  6. ProjDevBench
  7. Task Definition and Scope
  8. Model input.
  9. Model output.
  10. Problem Collection
  11. Stage I: Initial collection.
  12. Stage II: Scope-based filtering.
  13. Stage III: Quality-based filtering.
  14. Evaluation Protocol
  15. Execution-based evaluation.
  16. ...and 44 more sections

Figures (5)

  • Figure 1: Task comparison. Unlike benchmarks where coding agents modify code snippets from pre-existing codebases based on issues or pull requests, our benchmark evaluates end-to-end repository construction directly from project-level requirements.
  • Figure 2: Overview of the benchmark pipeline.
  • Figure 3: Problem collection and filtering.
  • Figure 4: Distribution of ProjDevBench tasks across 8 categories.
  • Figure 5: Human verification of LLM-based code review. Left: Correlation between the proposed LLM readability score and human scores. Right: Agreement on binary rule verification.