Tests as Prompt: A Test-Driven-Development Benchmark for LLM Code Generation

TL;DR

This paper addresses the challenge of evaluating code generation under test-driven development (TDD) by introducing WebApp1K, a benchmark of 1000 TDD tasks across 20 web-app domains implemented as React SPAs. Tests serve as both prompt and verification, and success is measured with $pass@k$ across up to $n=10$ attempts per task, highlighting the role of instruction following and in-context learning over raw coding proficiency. The authors provide a detailed error taxonomy with seven root causes and show that a large fraction of errors are singular or twin, indicating concentrated failure modes; a TLD experiment and a duo-feature upgrade explore the limits and scalability of TDD prompts under longer context. The work demonstrates that solving TDD tasks hinges on following coded instructions and leveraging in-context cues, and it outlines future directions to expand coverage, address instruction loss in long prompts, and solidify TDD as a practical, application-driven coding paradigm for LLMs.

Abstract

We introduce WebApp1K, a novel benchmark for evaluating large language models (LLMs) in test-driven development (TDD) tasks, where test cases serve as both prompt and verification for code generation. Unlike traditional approaches relying on natural language prompts, our benchmark emphasizes the ability of LLMs to interpret and implement functionality directly from test cases, reflecting real-world software development practices. Comprising 1000 diverse challenges across 20 application domains, the benchmark evaluates LLMs on their ability to generate compact, functional code under the constraints of context length and multi-feature complexity. Our findings highlight instruction following and in-context learning as critical capabilities for TDD success, surpassing the importance of general coding proficiency or pretraining knowledge. Through comprehensive evaluation of 19 frontier models, we reveal performance bottlenecks, such as instruction loss in long prompts, and provide a detailed error analysis spanning multiple root causes. This work underscores the practical value of TDD-specific benchmarks and lays the foundation for advancing LLM capabilities in rigorous, application-driven coding scenarios.

Figures (19)

• Figure 1: Incremental TDD by Human
• Figure 2: Transactional TDD by LLM
• Figure 3: Failures per problem
• Figure 4: Distribution of singular and twin errors
• Figure 5: Error distribution by models