Towards Human-AI Synergy in UI Design: Supporting Iterative Generation with LLMs

TL;DR

PrototypeFlow tackles the lack of iterative support in automated UI design by introducing a decoupled, multi-module system that uses a Theme Design Module and knowledge bases to generate editable SVG prototypes from natural language prompts and wireframes. It enables a top-down generation process with intermediate checkpoints and component-level sub-modules for text, image, and icon generation, guided by a retrieval-based knowledge base and an LLM controller to ensure design intent alignment. Quantitative evaluations show improved realism (lower FID) and diversity (higher GD) over baselines, while user studies demonstrate increased perceived usefulness, transparency, and control. The work highlights practical implications for human-AI collaboration in professional UI design and suggests future work on dynamic knowledge bases and cross-domain web design integration.

Abstract

In automated UI design generation, a key challenge is the lack of support for iterative processes, as most systems focus solely on end-to-end output. This stems from limited capabilities in interpreting design intent and a lack of transparency for refining intermediate results. To better understand these challenges, we conducted a formative study that identified concrete and actionable requirements for supporting iterative design with Generative Tools. Guided by these findings, we propose PrototypeFlow, a human-centered system for automated UI generation that leverages multi-modal inputs and models. PrototypeFlow takes natural language descriptions and layout preferences as input to generate the high-fidelity UI design. At its core is a theme design module that clarifies implicit design intent through prompt enhancement and orchestrates sub-modules for component-level generation. Designers retain full control over inputs, intermediate results, and final prototypes, enabling flexible and targeted refinement by steering generation and directly editing outputs. Our experiments and user studies confirmed the effectiveness and usefulness of our proposed PrototypeFlow.

Figures (11)

• Figure 1: Illustrative Examples of Formative Study and Key Findings
• Figure 2: Overview of PrototypeFlow System: This figure illustrates the main process of PrototypeFlow in response to a designer's input and UI wireframe. PrototypeFlow utilizes a multi-modal approach for the interactive generation of UI prototypes. It encompasses four specialized modules—Theme Design $M_{theme}$, Textual Content $M_{text}$, Image Content $M_{img}$, and Icon $M_{icon}$. The Theme Design Module $M_{theme}$ acts as the central coordinator, steering the collaborative efforts of the three sub-modules. By leveraging a cache pool, PrototypeFlow adeptly integrates the contributions from each module to ensure a cohesive alignment with the overall design context. This process not only generates accurate prototypes but also provides explainable intermediate results, enabling designers to conduct thorough reviews and make precise edits.
• Figure 3: Examples of SVG code renderings for icons and their corresponding semantic descriptions
• Figure 4: The prompt design for generating theme description. It consists of four parts: (A) Design Prompt, (B) Wireframe Layout, (C) Top-k Retrieved Knowledge Items, and (D) Theme Description Template. In the wireframe layout preview, different colours denote various component types as defined by Enrico leiva2020enrico.
• Figure 5: Layout-guided text-to-image Generation.