Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

UICopilot: Automating UI Synthesis via Hierarchical Code Generation from Webpage Designs

Yi Gui, Zhen Li, Zhongyi Zhang, Yao Wan, Dongping Chen, Hongyu Zhang, Yi Su, Bohua Chen, Xing Zhou, Wenbin Jiang, Xiangliang Zhang

TL;DR

UICopilot tackles UI synthesis from real webpage designs by decoupling structure prediction from fine-grained code generation. It uses a ViT-based Pix2Struct structure model to produce a coarse DOM layout with BBoxes, applies pruning to reduce noise, and then generates leaf-region HTML/CSS with segmented code followed by a global refinement step using GPT-4V. On WebCode2M, it outperforms strong baselines on automatic metrics and is consistently preferred in human evaluations, with the most pronounced gains on longer, more complex pages. The results demonstrate that hierarchical generation effectively mitigates the challenges of lengthy code and nested structures, enabling more accurate, design-faithful UI synthesis.

Abstract

Automating the synthesis of User Interfaces (UIs) plays a crucial role in enhancing productivity and accelerating the development lifecycle, reducing both development time and manual effort. Recently, the rapid development of Multimodal Large Language Models (MLLMs) has made it possible to generate front-end Hypertext Markup Language (HTML) code directly from webpage designs. However, real-world webpages encompass not only a diverse array of HTML tags but also complex stylesheets, resulting in significantly lengthy code. The lengthy code poses challenges for the performance and efficiency of MLLMs, especially in capturing the structural information of UI designs. To address these challenges, this paper proposes UICopilot, a novel approach to automating UI synthesis via hierarchical code generation from webpage designs. The core idea of UICopilot is to decompose the generation process into two stages: first, generating the coarse-grained HTML hierarchical structure, followed by the generation of fine-grained code. To validate the effectiveness of UICopilot, we conduct experiments on a real-world dataset, i.e., WebCode2M. Experimental results demonstrate that UICopilot significantly outperforms existing baselines in both automatic evaluation metrics and human evaluations. Specifically, statistical analysis reveals that the majority of human annotators prefer the webpages generated by UICopilot over those produced by GPT-4V.

UICopilot: Automating UI Synthesis via Hierarchical Code Generation from Webpage Designs

TL;DR

UICopilot tackles UI synthesis from real webpage designs by decoupling structure prediction from fine-grained code generation. It uses a ViT-based Pix2Struct structure model to produce a coarse DOM layout with BBoxes, applies pruning to reduce noise, and then generates leaf-region HTML/CSS with segmented code followed by a global refinement step using GPT-4V. On WebCode2M, it outperforms strong baselines on automatic metrics and is consistently preferred in human evaluations, with the most pronounced gains on longer, more complex pages. The results demonstrate that hierarchical generation effectively mitigates the challenges of lengthy code and nested structures, enabling more accurate, design-faithful UI synthesis.

Abstract

Automating the synthesis of User Interfaces (UIs) plays a crucial role in enhancing productivity and accelerating the development lifecycle, reducing both development time and manual effort. Recently, the rapid development of Multimodal Large Language Models (MLLMs) has made it possible to generate front-end Hypertext Markup Language (HTML) code directly from webpage designs. However, real-world webpages encompass not only a diverse array of HTML tags but also complex stylesheets, resulting in significantly lengthy code. The lengthy code poses challenges for the performance and efficiency of MLLMs, especially in capturing the structural information of UI designs. To address these challenges, this paper proposes UICopilot, a novel approach to automating UI synthesis via hierarchical code generation from webpage designs. The core idea of UICopilot is to decompose the generation process into two stages: first, generating the coarse-grained HTML hierarchical structure, followed by the generation of fine-grained code. To validate the effectiveness of UICopilot, we conduct experiments on a real-world dataset, i.e., WebCode2M. Experimental results demonstrate that UICopilot significantly outperforms existing baselines in both automatic evaluation metrics and human evaluations. Specifically, statistical analysis reveals that the majority of human annotators prefer the webpages generated by UICopilot over those produced by GPT-4V.
Paper Structure (19 sections, 2 equations, 10 figures, 2 tables)

This paper contains 19 sections, 2 equations, 10 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Hierarchy Structure of Webpage Designs
  4. Image to Code by MLLMs
  5. UICopilot
  6. Task Description
  7. Overview
  8. Coarse DOM Layout Generation
  9. Fine-Grained Code Synthesis
  10. Experiments and Analysis
  11. Datasets
  12. Evaluation Metrics
  13. Baselines
  14. The Effectiveness of UICopilot
  15. The Influence of min_area and max_depth
  16. ...and 4 more sections

Figures (10)

  • Figure 1: The structural information of a webpage.
  • Figure 2: An overview of our proposed UICopilot, which is composed of three components: (a) coarse DOM layout generation, (b) leaf node HTML/CSS generation, and (c) non-leaf style synthesis and global refinement.
  • Figure 3: An example of DOM tree in JSON format.
  • Figure 4: BBox-based data pruning.
  • Figure 5: The prompt for leaf node HTML/CSS generation.
  • ...and 5 more figures