ABC-Bench: Benchmarking Agentic Backend Coding in Real-World Development

TL;DR

ABC-Bench targets a practical gap in evaluating LLM-driven agents by testing full, end-to-end backend development tasks that include repository exploration, environment provisioning, container deployment, and API-level verification. It introduces the ABC-Pipeline to automatically mine real-world open-source repositories, synthesize deployable environments, and produce masked, solvable tasks across 8 languages and 19 frameworks (224 tasks total, 92 with autonomous environment configuration). Extensive experiments show state-of-the-art models still struggle with reliability on holistic backend tasks, with environment configuration and deployment as the principal bottlenecks and a strong correlation between interaction depth and success (r = 0.87). The results highlight a significant gap between current agent capabilities and real-world backend engineering demands, and the work provides open-source benchmarks and baselines to guide future improvements in agentic software production systems.

Abstract

The evolution of Large Language Models (LLMs) into autonomous agents has expanded the scope of AI coding from localized code generation to complex, repository-level, and execution-driven problem solving. However, current benchmarks predominantly evaluate code logic in static contexts, neglecting the dynamic, full-process requirements of real-world engineering, particularly in backend development which demands rigorous environment configuration and service deployment. To address this gap, we introduce ABC-Bench, a benchmark explicitly designed to evaluate agentic backend coding within a realistic, executable workflow. Using a scalable automated pipeline, we curated 224 practical tasks spanning 8 languages and 19 frameworks from open-source repositories. Distinct from previous evaluations, ABC-Bench require the agents to manage the entire development lifecycle from repository exploration to instantiating containerized services and pass the external end-to-end API tests. Our extensive evaluation reveals that even state-of-the-art models struggle to deliver reliable performance on these holistic tasks, highlighting a substantial disparity between current model capabilities and the demands of practical backend engineering. Our code is available at https://github.com/OpenMOSS/ABC-Bench.

Figures (8)

• Figure 1: Overview of the ABC-Bench evaluation pipeline. The figure illustrates the closed-loop evaluation process. In the development phase (Steps 1--4), the agent acts as a backend expert to analyze the repository, resolve issues, and draft container specifications. Transitioning to the validation phase, the benchmark system builds a Docker image from the agent's output and deploys the service (Step 5). Finally, the functional correctness is verified by sending real HTTP requests to the deployed endpoint (Step 6), ensuring the fix works in a production-like environment.
• Figure 2: Overview of the ABC-Bench dataset composition. The left pie chart illustrates the distribution of tasks across eight major programming languages. The right chart provides a detailed breakdown of the 19 web frameworks involved, demonstrating the benchmark's capability to evaluate models across a wide spectrum of real-world software stacks.
• Figure 3: Overview of the ABC-Pipeline workflow. The process consists of three phases: (1) Repository Exploration, where backend repositories are filtered and API tests are generated; (2) Environment Setup & Verification, which involves synthesizing Docker environments and verifying them against generated tests; and (3) Task Instantiation, where the final benchmark task is constructed by applying git patches and generating task instructions.
• Figure 4: Analysis of environment configuration capabilities. Comparison of various models (including Claude Sonnet 4.5, GPT-5, DeepSeek-V3.2, and Qwen3-8B) across 92 environment-related tasks. The bar charts display Build Success ($S_1$) and Conditional End-to-End Success ($S_2$). The red line plot indicates the Average Pass@1.
• Figure 5: Interaction turns vs. performance. Scatter plot illustrating the relationship between the average number of agent turns (x-axis) and the Average Pass@1 rate (y-axis) across various models. The blue fitted trend line reveals a strong positive correlation ($r=0.87$).