Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ArtifactsBench: Bridging the Visual-Interactive Gap in LLM Code Generation Evaluation

Chenchen Zhang, Yuhang Li, Can Xu, Jiaheng Liu, Ao Liu, Changzhi Zhou, Ken Deng, Dengpeng Wu, Guanhua Huang, Kejiao Li, Qi Yi, Ruibin Xiong, Shihui Hu, Yue Zhang, Yuhao Jiang, Zenan Xu, Yuanxing Zhang, Wiggin Zhou, Chayse Zhou, Fengzong Lian

TL;DR

ArtifactsBench tackles the urgent need to evaluate LLMs on their ability to generate high-quality interactive visual artifacts. It introduces a large-scale, automated benchmark that uses deterministic sandbox execution, staged visual evidence, and checklist-guided multimodal referees to assess visual fidelity, interactivity, and code quality. The framework achieves strong alignment with human preferences, demonstrating robustness across 30+ models and revealing that instruction-tuned generalists often outperform domain-specific systems. By open-sourcing the benchmark, evaluation harness, and baselines, ArtifactsBench provides a scalable tool to drive progress toward user-centric visual-code generation.

Abstract

The generative capabilities of Large Language Models (LLMs) are rapidly expanding from static code to dynamic, interactive visual artifacts. This progress is bottlenecked by a critical evaluation gap: established benchmarks focus on algorithmic correctness and are blind to the visual fidelity and interactive integrity that define modern user experiences. To bridge this gap, we introduce ArtifactsBench, a new benchmark and paradigm for the automated, multimodal evaluation of visual code generation. Our framework programmatically renders each generated artifact and captures its dynamic behavior through temporal screenshots. This visual evidence, alongside the source code, is then assessed by a Multimodal LLM (MLLM)-as-Judge, which is rigorously guided by a fine-grained, per-task checklist to ensure holistic and reproducible scoring. We construct a new benchmark of 1,825 diverse tasks and evaluate over 30 leading LLMs. Our automated evaluation achieves a striking 94.4% ranking consistency with WebDev Arena, the gold-standard for human preference in web development, and over 90% pairwise agreement with human experts. This establishes ArtifactsBench as the first framework to reliably automate the assessment of human-perceived quality at scale. Our analysis provides a high-resolution map of the current SOTA, revealing that generalist models often outperform domain-specific ones. We open-source ArtifactsBench, including the benchmark, evaluation harness, and baseline results at https://artifactsbenchmark.github.io/, to provide the community with a scalable and accurate tool to accelerate the development of user-centric generative models.

ArtifactsBench: Bridging the Visual-Interactive Gap in LLM Code Generation Evaluation

TL;DR

ArtifactsBench tackles the urgent need to evaluate LLMs on their ability to generate high-quality interactive visual artifacts. It introduces a large-scale, automated benchmark that uses deterministic sandbox execution, staged visual evidence, and checklist-guided multimodal referees to assess visual fidelity, interactivity, and code quality. The framework achieves strong alignment with human preferences, demonstrating robustness across 30+ models and revealing that instruction-tuned generalists often outperform domain-specific systems. By open-sourcing the benchmark, evaluation harness, and baselines, ArtifactsBench provides a scalable tool to drive progress toward user-centric visual-code generation.

Abstract

The generative capabilities of Large Language Models (LLMs) are rapidly expanding from static code to dynamic, interactive visual artifacts. This progress is bottlenecked by a critical evaluation gap: established benchmarks focus on algorithmic correctness and are blind to the visual fidelity and interactive integrity that define modern user experiences. To bridge this gap, we introduce ArtifactsBench, a new benchmark and paradigm for the automated, multimodal evaluation of visual code generation. Our framework programmatically renders each generated artifact and captures its dynamic behavior through temporal screenshots. This visual evidence, alongside the source code, is then assessed by a Multimodal LLM (MLLM)-as-Judge, which is rigorously guided by a fine-grained, per-task checklist to ensure holistic and reproducible scoring. We construct a new benchmark of 1,825 diverse tasks and evaluate over 30 leading LLMs. Our automated evaluation achieves a striking 94.4% ranking consistency with WebDev Arena, the gold-standard for human preference in web development, and over 90% pairwise agreement with human experts. This establishes ArtifactsBench as the first framework to reliably automate the assessment of human-perceived quality at scale. Our analysis provides a high-resolution map of the current SOTA, revealing that generalist models often outperform domain-specific ones. We open-source ArtifactsBench, including the benchmark, evaluation harness, and baseline results at https://artifactsbenchmark.github.io/, to provide the community with a scalable and accurate tool to accelerate the development of user-centric generative models.

Paper Structure

This paper contains 41 sections, 17 figures, 7 tables.

Table of Contents

  1. Introduction
  2. ArtifactsBench: A Benchmark for Visual Code Generation
  3. Overview
  4. Dataset Construction Pipeline
  5. Evaluation Methodology
  6. Fine-Grained Checklists
  7. Automated Evaluation and Referees
  8. Experiments
  9. Settings
  10. Main Results and Analysis
  11. Proprietary multimodal models show clear advantage.
  12. Performance scales with model size and deliberation time.
  13. Analysis of open-source model performance.
  14. Opportunities remain in challenging scenarios.
  15. Generalist skills outperform specialist expertise.
  16. ...and 26 more sections

Figures (17)

  • Figure 1: Distribution of ArtifactsBench
  • Figure 2: An overview of the data construction process of ArtifactsBench.
  • Figure 3: The ArtifactsBench evaluation pipeline. The process hinges on a two-stage evaluation: (Step 5) we first validate our MLLM-as-Judge by confirming its high pairwise scoring agreement with human experts on a controlled set of tasks. (Step 6) Once its reliability is established, the automated judge is deployed at scale to evaluate all model outputs across the entire benchmark. The final rankings are then cross-validated against WebDev Arena to ensure alignment with real-world user preferences for visual quality.
  • Figure 4: Correlation between vision- and code-oriented scores on ArtifactsBench. The strong positive trend indicates holistic capability and motivates assessing both dimensions.
  • Figure 5: Scores across difficulty tiers on ArtifactsBench.
  • ...and 12 more figures