WebDevJudge: Evaluating (M)LLMs as Critiques for Web Development Quality

TL;DR

WebDevJudge introduces a comprehensive meta-evaluation benchmark to assess LLM-based judges on dynamic, interactive web development tasks. It combines static and interactive evaluation with rubric-grounded ground truth, refined data filtering, and rubric-tree annotations to produce high-quality human-preference labels. The study reveals a persistent gap between current LLM judges and human experts, driven by biases, difficulty recognizing functional equivalence, and limitations in feasibility verification, especially in open-ended, interactive environments. It also provides a diagnostic WebDevJudge-Unit dataset and insights into evaluation protocols, guiding future work toward more reliable automated evaluators for complex tasks.

Abstract

The paradigm of LLM-as-a-judge is emerging as a scalable and efficient alternative to human evaluation, demonstrating strong performance on well-defined tasks. However, its reliability in open-ended tasks with dynamic environments and complex interactions remains unexplored. To bridge the gap, we introduce WebDevJudge, a systematic benchmark for assessing LLM-as-a-judge performance in web development, with support for both non-interactive evaluation based on static observations and continuous interactive evaluation with a dynamic web environment. WebDevJudge comprises human preference labels over paired web implementations, annotated with structured and query-grounded rubrics to ensure high-quality ground truth. Using this benchmark, we comprehensively evaluate various evaluators, including LLMs, MLLMs, and agentic workflows. We systematically investigate the impact of different paradigms and guidance mechanisms. Our experiments reveal a significant gap between LLM judges and human experts. In-depth analysis indicates this gap stems from fundamental model limitations, including failures in recognizing functional equivalence, verifying task feasibility, and mitigating bias. Overall, WebDevJudge presents a significant challenge to LLM-as-a-judge, offering insights to guide future research toward developing more reliable and capable automated evaluators for complicated scenarios. Code and data are available at https://github.com/lcy2723/WebDevJudge.

Figures (7)

• Figure 1: Overview of WebDevJudge. Left: Data Collection with filtering. Center: Annotation with Rubric. Right: Evaluate various evaluators based on different evaluation paradigms.
• Figure 2: The distribution of sub-category across WebDevJudge.
• Figure 3: Agreement rates of LLM evaluators under various guidance protocols. We compare pairwise and single-answer paradigms using direct judgment, Likert scales, and structured rubrics.
• Figure 4: An example illustrating the failure to recognize functional equivalence. The webpage element for 'Presentation' serves the same purpose as the required 'Demonstration' rating.
• Figure 5: The consistency between model predictions under pairwise comparison.