Bridging Gulfs in UI Generation through Semantic Guidance

TL;DR

This work tackles the challenges of building UI designs with generative AI by introducing an explicit semantic intermediate representation that sits between human intent and AI output. A four-level hierarchical framework (Product, Design System, Feature, Component) is derived from prompting guidelines and used to drive a semantic-based UI generation system offering structured input, augmented output analysis, and relationship-aware iterative refinement. In a comparative study with 14 practitioners, the semantic system significantly improved intent expressiveness, output interpretability, and refinement control, suggesting a shift from trial-and-error prompting toward systematic, explainable design exploration. The approach enhances transparency and traceability in AI-driven UI design, with implications for better collaboration, trust, and efficiency in real-world design workflows, while outlining avenues for integration with design systems and scalability to larger UI architectures.

Abstract

While generative AI enables high-fidelity UI generation from text prompts, users struggle to articulate design intent and evaluate or refine results-creating gulfs of execution and evaluation. To understand the information needed for UI generation, we conducted a thematic analysis of UI prompting guidelines, identifying key design semantics and discovering that they are hierarchical and interdependent. Leveraging these findings, we developed a system that enables users to specify semantics, visualize relationships, and extract how semantics are reflected in generated UIs. By making semantics serve as an intermediate representation between human intent and AI output, our system bridges both gulfs by making requirements explicit and outcomes interpretable. A comparative user study suggests that our approach enhances users' perceived control over intent expression, outcome interpretation, and facilitates more predictable, iterative refinement. Our work demonstrates how explicit semantic representation enables systematic and explainable exploration of design possibilities in AI-driven UI design.

Figures (6)

• Figure 1: From prompting guidelines to a semantic framework. We conducted a thematic analysis of prompting guidelines from major UI generation tools, surfacing recurring patterns of what information users specify. This yields a four-level hierarchical representation — , , , — that organizes interdependent design semantics from high- to low-levels. Semantic elements are related both vertically (between levels) and horizontally (within levels), meaning that changes to one element can cascade across others.
• Figure 2: System interface with semantic input, generated UI, and analysis views. Top: Users specify semantics across the four-level framework (left), which are used to generate the corresponding UI (right). Bottom: The Relation View visualizes dependencies among semantic attributes, while the Semantic Detail View shows a selected attribute with its values, upstream and downstream relations, and suggested refinements for coherent design decisions.
• Figure 3: Example of input semantics specification. Users can define semantics manually or via natural language prompts. Natural language input is parsed into structured semantics across our four-level framework: ---Description, ---Design Style, Color, ---Function, and ---Card Component.
• Figure 4: System workflow from input semantics to augmented semantics. Input semantics provided by the user (left) are transformed into a functional UI through the Generate step (center). When Analyze is triggered, the system augments this view with additional semantics inferred from the UI (right).
• Figure 5: Example of semantic relationship analysis. A mobile shopping app specification is analyzed to reveal dependencies between semantic attributes. The system visualizes relations as a graph, where arrows denote influences between attributes: green edges indicate Values Match Well, orange edges highlight Values Conflict, and blue dashed edges denote Needs Value.