E2Edev: Benchmarking Large Language Models in End-to-End Software Development Task

TL;DR

E2EDev presents a Behavior-Driven Development–based benchmark for End-to-End Software Development (E2ESD), addressing limitations of prior coarse, loosely specified benchmarks by introducing fine-grained user requirements, executable BDD tests, and a HITL-MAA annotation framework. Across diverse web-app projects and multiple LLM backbones, the study demonstrates that current models struggle with project-level E2ESD, often achieving only 30–60% requirement fidelity and incurring high costs in multi-agent setups, highlighting the need for more effective and cost-efficient approaches. The benchmark provides a rigorous, standardized evaluation framework with transparent metrics (Req.Acc, Test.Acc, Balanced Score) and a reproducible pipeline using Behave, enabling objective comparisons and guiding future improvements in automated end-to-end software construction. By releasing the codebase and dataset publicly, E2EDev aims to catalyze ongoing progress in harnessing LLMs for practical software engineering tasks while promoting responsible, scalable benchmarking practices.

Abstract

The rapid advancement in large language models (LLMs) has demonstrated significant potential in End-to-End Software Development (E2ESD). However, existing E2ESD benchmarks are limited by coarse-grained requirement specifications and unreliable evaluation protocols, hindering a true understanding of current framework capabilities. To address these limitations, we present E2EDev, a novel benchmark grounded in the principles of Behavior-Driven Development (BDD), which evaluates the capabilities of E2ESD frameworks by assessing whether the generated software meets user needs through mimicking real user interactions (Figure 1). E2EDev comprises (i) a fine-grained set of user requirements, (ii) multiple BDD test scenarios with corresponding Python step implementations for each requirement, and (iii) a fully automated testing pipeline built on the Behave framework. To ensure its quality while reducing the annotation effort, E2EDev leverages our proposed Human-in-the-Loop Multi-Agent Annotation Framework (HITL-MAA). By evaluating various E2ESD frameworks and LLM backbones with E2EDev, our analysis reveals a persistent struggle to effectively solve these tasks, underscoring the critical need for more effective and cost-efficient E2ESD solutions. Our codebase and benchmark are publicly available at https://github.com/SCUNLP/E2EDev.

Figures (11)

• Figure 2: Comparing existing work with ours, existing benchmarks use coarse-grained requirements with unreliable evaluations whereas E2ESD provides fine-grained requirements with executable BDD tests.
• Figure 3: HITL-MAA framework for semi-automated dataset construction. Given source code, the framework extracts user requirements and generates corresponding BDD test scenarios (Gherkin format) along with Python step implementations, with human verification or agent-based refinement at each stage to ensure dataset quality.
• Figure 4: Soft Req. Acc. and Req. Acc. under three representative LLMs. Here, V-LLM refers to the Vanilla LLM, GPT-E refers to GPT-Engineer, and Self-C refers to Self-Collaboration. Additional results on other backbones are reported in Appendix \ref{['appe2e']}, Table \ref{['app_tb:soft']}.
• Figure 5: Comparisons across agentic frameworks and the Vanilla LLM (on Req. Acc.). Bars within the gray zone indicate lower performance.
• Figure 6: Statistics of interaction overhead across representative frameworks. P. Tok., C. Tok., and Turns refer to average prompt tokens, completion tokens, and dialogue rounds, respectively.