Benchmarking and Studying the LLM-based Agent System in End-to-End Software Development

TL;DR

This paper tackles the evaluation gap for LLM-based autonomous agents in end-to-end software development by introducing E2EDevBench, a dynamically curated, realistic benchmark, and a hybrid evaluation framework that combines automated test migration with LLM-based requirement verification. Using a unified SWE-Agent foundation, it conducts a controlled empirical study of three workflow configurations to isolate the effects of architecture and orchestration. The findings show that state-of-the-art agents can fulfill roughly $50.0\%$ of requirements on the benchmark, with performance highly dependent on task decomposition and collaboration strategies, and reveal that the main bottlenecks lie in requirement understanding and self-verification rather than coding capability. These insights, along with the benchmark and evaluation framework, offer practical guidance for advancing requirement comprehension, planning, and robust agent design in real-world software development.

Abstract

The development of LLM-based autonomous agents for end-to-end software development represents a significant paradigm shift in software engineering. However, the scientific evaluation of these systems is hampered by significant challenges, including overly simplistic benchmarks and the difficulty of conducting fair comparisons between different agent architectures due to confounding implementation variables. To address these limitations, we first construct a challenging and dynamically curated E2EDevBench to simulate realistic development scenarios. Second, we propose a hybrid evaluation framework that combines test-case-based functional assessment with fine-grained, LLM-based requirement verification. Using this framework, we conduct a controlled empirical study on three representative agent architectures implemented upon a unified foundation to isolate the impact of workflow design. Our findings reveal that state-of-the-art agents can fulfill approximately 50\% of requirements on \bench{}, but their success is critically dependent on the architectural strategy for task decomposition and collaboration. Furthermore, our analysis indicates that the primary bottleneck is the omission of requirements and inadequate self-verification. This work provides the community with a more realistic benchmark, a comprehensive evaluation framework, and crucial insights into the current capabilities and core challenges of software development agents, guiding future research toward enhancing requirement comprehension and planning.

Figures (5)

• Figure 1: The dataset construction and evaluation workflow of E2EDevBench.
• Figure 2: Analysis of agreement between LLM-as-Judge and human experts.
• Figure 3: Agent performance stability across different time periods.
• Figure 4: Distribution of first test writing step by agent.
• Figure 5: Implemented requirements comparison across 3 runs for each agent.