WebApp1K: A Practical Code-Generation Benchmark for Web App Development

TL;DR

WebApp1K addresses the need for a practical, end-to-end benchmark to evaluate how well LLMs can generate functional web apps. The authors design a React-based benchmark with 1000 user-journey problems, validated by fetchMock test sequences and assessed via pass@k metrics to capture end-to-end correctness. Their findings show strong performance from open-source LLM families, a clear correlation between model size and code correctness, and no universal prompting technique that consistently improves results across all models. The work provides an accessible, market-oriented evaluation framework and a leaderboard to guide future improvements in practical web app code generation. This benchmark has potential to influence model selection and development efforts aimed at deployable software prototyping with LLMs.

Abstract

We introduce WebApp1K, a practical code-generation benchmark to measure LLM ability to develop web apps. This benchmark aims to calibrate LLM output and aid the models to progressively improve code correctness and functionality. The benchmark is lightweight and easy to run. We present the initial version of WebApp1K, and share our findings of running the benchmark against the latest frontier LLMs. First, open source LLMs deliver impressive performance, closely trailing behind GPT-4o and Claude 3.5. Second, model size has strong correlation with code correctness. Third, no prompting techniques have been found to lift performance either universally to all models, or significantly to a single model.