Misty: UI Prototyping Through Interactive Conceptual Blending

TL;DR

Misty introduces an interactive UI prototyping workflow based on conceptual blending to integrate design examples into ongoing UI code. The system uses a node-based infinite canvas, global and targeted blending, and semantic diffs, powered by a multimodal AI model (GPT-4o). In an exploratory first-use study with 14 frontend developers, Misty kickstarts ideation, supports flexible intent specification, and inspires serendipitous blends, while exposing current AI limitations. The work demonstrates a path toward tools that blur designer–developer boundaries and accelerate iterative UI prototyping with responsible AI considerations.

Abstract

UI prototyping often involves iterating and blending elements from examples such as screenshots and sketches, but current tools offer limited support for incorporating these examples. Inspired by the cognitive process of conceptual blending, we introduce a novel UI workflow that allows developers to rapidly incorporate diverse aspects from design examples into work-in-progress UIs. We prototyped this workflow as Misty. Through an exploratory first-use study with 14 frontend developers, we assessed Misty's effectiveness and gathered feedback on this workflow. Our findings suggest that Misty's conceptual blending workflow helps developers kickstart creative explorations, flexibly specify intent in different stages of prototyping, and inspires developers through serendipitous UI blends. Misty demonstrates the potential for tools that blur the boundaries between developers and designers.

Figures (2)

• Figure 1: Event trace of all blending operations and user evaluation scores during the user study sessions. Each circle represents a blending operation, laid out from left to right based on the time it occurred during the study. The number inside is the participant's evaluation of how well the blending output matched expectations, with 1 as not satisfied and 5 very satisfied. Based on these evaluation scores, we also varied the circles' vertical positions. The color of each circle represents the type of operation in Misty: Global blend, Drag and drop, Semantic diff, Regenerate, Code edit. We normalize the space between operations.
• Figure 2: The usability questionnaire results from our user study. User satisfaction survey results for Misty: most participants found Misty easy to use and would like to use it in their work regularly. Misty is generally well-received, with potential for targeted improvements in user intent interpretation and interface refinement.