Widget2Code: From Visual Widgets to UI Code via Multimodal LLMs

TL;DR

This work formalizes Widget2Code, a challenging UI2Code task focused on compact, icon-rich widgets that lack accessible markup. It introduces an image-only widget benchmark with fine-grained metrics spanning layout, legibility, style, and geometry, and provides a comprehensive baseline (Perceptual Agent + WidgetFactory) that couples perceptual decomposition with a DSL-driven, multi-framework compiler. Benchmark results reveal that while general-purpose MLLMs outperform UI2Code-specialists on widgets, substantial gaps remain in layout fidelity and stylistic accuracy, motivating the proposed modular baseline. The framework enables controlled reconstruction, geometry-aware rendering, and supports future research through synthetic data generation via WidgetFactory and supervised fine-tuning on widget-focused tasks. Overall, the work lays a unified foundation for reliable, interpretable, and cross-platform Widget2Code development and benchmarking.

Abstract

User interface to code (UI2Code) aims to generate executable code that can faithfully reconstruct a given input UI. Prior work focuses largely on web pages and mobile screens, leaving app widgets underexplored. Unlike web or mobile UIs with rich hierarchical context, widgets are compact, context-free micro-interfaces that summarize key information through dense layouts and iconography under strict spatial constraints. Moreover, while (image, code) pairs are widely available for web or mobile UIs, widget designs are proprietary and lack accessible markup. We formalize this setting as the Widget-to-Code (Widget2Code) and introduce an image-only widget benchmark with fine-grained, multi-dimensional evaluation metrics. Benchmarking shows that although generalized multimodal large language models (MLLMs) outperform specialized UI2Code methods, they still produce unreliable and visually inconsistent code. To address these limitations, we develop a baseline that jointly advances perceptual understanding and structured code generation. At the perceptual level, we follow widget design principles to assemble atomic components into complete layouts, equipped with icon retrieval and reusable visualization modules. At the system level, we design an end-to-end infrastructure, WidgetFactory, which includes a framework-agnostic widget-tailored domain-specific language (WidgetDSL) and a compiler that translates it into multiple front-end implementations (e.g., React, HTML/CSS). An adaptive rendering module further refines spatial dimensions to satisfy compactness constraints. Together, these contributions substantially enhance visual fidelity, establishing a strong baseline and unified infrastructure for future Widget2Code research.

Figures (23)

• Figure 1: Comparison across interface modalities. Web and mobile UIs provide rich structural and textual context that supports rule-based code mapping, whereas widgets employ dense iconography, embedded graphs, and vivid color schemes within highly constrained layouts. These stylistic and structural compactness factors pose substantial challenges for UI-to-Code reconstruction.
• Figure 2: Style score comparison on our widget benchmark. Generalized MLLMs outperform specialized UI2Code models, which are tuned for other UI formats instead of widgets.
• Figure 3: Qualitative comparison of existing methods on the Widget2Code benchmark. Specialized UI2Code models fail to reproduce the input appearance, while generalized MLLMs generate more coherent results. However, all models still exhibit issues such as content overflow, structural misalignment, and color inconsistencies, highlighting the challenges of faithful widget reconstruction.
• Figure 4: A visual example of a challenging case. Failure cases in generalized MLLMs commonly arise from layout misinterpretation, content occlusion, missing components, and difficulty reconstructing complex graphs.
• Figure 5: Overview of our Widget2Code framework. The pipeline consists of three components: (1) data curation, which collects and processes widget images to construct our benchmark; (2) the Perceptual Agent, which decomposes the input into atomic components and extracts visual, semantic, and stylistic cues; and (3) WidgetFactory, an end-to-end infrastructure that generates, compiles, and adaptively renders WidgetDSL to reconstruct the input widget.