WebCoderBench: Benchmarking Web Application Generation with Comprehensive and Interpretable Evaluation Metrics

TL;DR

WebCoderBench tackles the challenge of evaluating LLM generated web apps by leveraging 1,572 real world requirements and a 24 metric suite across 9 perspectives. It blends rule based metrics with LLM judged assessments and uses ground truth aligned weights to produce interpretable overall scores that reflect user preferences. Experiments with 12 LLMs and 2 agents reveal no single model dominates all aspects and show a narrowing gap between open and closed models, highlighting opportunities for targeted improvements. The benchmark promises practical impact by guiding model development toward user prioritized quality attributes and providing a framework for ongoing, interpretable evaluation.

Abstract

Web applications (web apps) have become a key arena for large language models (LLMs) to demonstrate their code generation capabilities and commercial potential. However, building a benchmark for LLM-generated web apps remains challenging due to the need for real-world user requirements, generalizable evaluation metrics without relying on ground-truth implementations or test cases, and interpretable evaluation results. To address these challenges, we introduce WebCoderBench, the first real-world-collected, generalizable, and interpretable benchmark for web app generation. WebCoderBench comprises 1,572 real user requirements, covering diverse modalities and expression styles that reflect realistic user intentions. WebCoderBench provides 24 fine-grained evaluation metrics across 9 perspectives, combining rule-based and LLM-as-a-judge paradigm for fully automated, objective, and general evaluation. Moreover, WebCoderBench adopts human-preference-aligned weights over metrics to yield interpretable overall scores. Experiments across 12 representative LLMs and 2 LLM-based agents show that there exists no dominant model across all evaluation metrics, offering an opportunity for LLM developers to optimize their models in a targeted manner for a more powerful version.

Figures (15)

• Figure 1: The dataset construction process of WebCoderBench.
• Figure 2: An example user requirement with its corresponding ground-truth checklists.
• Figure 3: The weight assignment and evaluation workflow of WebCoderBench.
• Figure 4: The weight proportion of each perspective and each evaluation metric.
• Figure 5: The detailed raw scores of 24 evaluation metrics for each LLM and LLM-based agent, with the x-axis indices denoting the IDs of evaluation metrics (corresponding to Table \ref{['metrics table']}), and the y-axis indices denoting the IDs of models (corresponding to Table \ref{['Main results table']}).