Interaction2Code: Benchmarking MLLM-based Interactive Webpage Code Generation from Interactive Prototyping

TL;DR

This work defines the Interaction-to-Code task and introduces Interaction2Code, the first benchmark for generating interactive webpages from interactive prototyping. It reveals four major limitations of current MLLMs in handling interactions, and proposes four enhancement strategies—Interactive Element Highlighting, Failure-aware Prompting, Visual Saliency Enhancement, and Visual-Textual Descriptions—to close the gap. Through a large, diverse dataset of 127 webpages and 374 interactions, evaluated across commercial and open-source models with both automatic and human assessments, the study shows that interactive parts are harder for models and that multimodal prompts significantly improve performance. The results offer practical guidance for improving MLLM-based UI code generation and point to future research directions toward full-stack and multi-page website generation.

Abstract

Multimodal Large Language Models (MLLMs) have demonstrated remarkable performance on the design-to-code task, i.e., generating UI code from UI mock-ups. However, existing benchmarks only contain static web pages for evaluation and ignore the dynamic interaction, limiting the practicality, usability and user engagement of the generated webpages. To bridge these gaps, we present the first systematic investigation of MLLMs in generating interactive webpages. Specifically, we formulate the Interaction-to-Code task and establish the Interaction2Code benchmark, encompassing 127 unique webpages and 374 distinct interactions across 15 webpage types and 31 interaction categories. Through comprehensive experiments utilizing state-of-the-art (SOTA) MLLMs, evaluated via both automatic metrics and human assessments, we identify four critical limitations of MLLM on Interaction-to-Code task: (1) inadequate generation of interaction compared with full page, (2) prone to ten types of failure, (3) poor performance on visually subtle interactions, and (4) insufficient undestanding on interaction when limited to single-modality visual descriptions. To address these limitations, we propose four enhancement strategies: Interactive Element Highlighting, Failureaware Prompting (FAP), Visual Saliency Enhancement, and Visual-Textual Descriptions Combination, all aiming at improving MLLMs' performance on the Interaction-toCode task. The Interaction2Code benchmark and code are available in https://github. com/WebPAI/Interaction2Code.

Figures (14)

• Figure 1: Interaction example and interactive elements ratio of different types of webpages.
• Figure 2: UI2Code.
• Figure 3: The construction of Interaction2Code benchmark.
• Figure 4: Topic and framework distribution.
• Figure 5: A higher usable rate indicates better functionality and a higher win rate indicates better quality.