Automatically Benchmarking LLM Code Agents through Agent-Driven Annotation and Evaluation

TL;DR

PRDBench introduces a PRD-centered, project-level benchmark for code agents that dramatically lowers annotation cost through an agent-driven data production pipeline. By combining scaffold and criteria generation with an EvalAgent-based scoring system, PRDBench supports flexible evaluation beyond unit tests, including shell interactions and file-level checks across 50 Python tasks in 20 domains. Empirical results show substantial annotation efficiency gains, reliable EvalAgent alignment with human judges, and meaningful differentiation among minimal and commercial code agents, while also highlighting cost and stability considerations in debugging. The framework enables scalable, real-world evaluation of AI-driven software development tools with broad applicability to future agent architectures and QA protocols.

Abstract

Recent advances in code agents have enabled automated software development at the project level, supported by large language models (LLMs) and widely adopted tools. However, existing benchmarks for code agent evaluation face two major limitations: high annotation cost and expertise requirements, and rigid evaluation metrics that rely primarily on unit tests. To address these challenges, we propose an agent-driven benchmark construction pipeline that leverages human supervision to efficiently generate diverse and challenging project-level tasks. Based on this approach, we introduce PRDBench, a novel benchmark comprising 50 real-world Python projects across 20 domains, each with structured Product Requirement Document (PRD) requirements, comprehensive evaluation criteria, and reference implementations. PRDBench features rich data sources, high task complexity, and flexible metrics. We further employ an Agent-as-a-Judge paradigm to score agent outputs, enabling the evaluation of various test types beyond unit tests. Extensive experiments on PRDBench demonstrate its effectiveness in assessing the capabilities of both code agents and evaluation agents, providing a scalable and robust framework for annotation and evaluation.

Figures (6)

• Figure 1: Overview of the PRDBench data production workflow. Step1: PRD and Test Plan Initialization; Step 2: Code Scaffold and Criteria Scheme Generation; Step 3: Human Inspection; Step 4: Agent-based Fix and Refinement; Step 5: Remove scaffold.
• Figure 2: A task example of PRDBench.
• Figure 3: Overview of PRDBench evaluation. The EvalAgent executes tests based on the criteria scheme using various tools, compares outputs (files or results) with expected outputs, and generates a report for the submitted code.
• Figure 4: Data statistics of PRDBench.
• Figure 5: Error rates of code agents on different types of test cases.