Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Flowy: Supporting UX Design Decisions Through AI-Driven Pattern Annotation in Multi-Screen User Flows

Yuwen Lu, Ziang Tong, Qinyi Zhao, Yewon Oh, Bryan Wang, Toby Jia-Jun Li

TL;DR

Flowy is an app that augments designers' information foraging process in ideation by supplementing specific user flow examples with distilled design pattern knowledge, and embodies a human-centered, intelligence augmentation approach to using AI in UX design.

Abstract

Many recent AI-powered UX design tools focus on generating individual static UI screens from natural language. However, they overlook the crucial aspect of interactions and user experiences across multiple screens. Through formative studies with UX professionals, we identified limitations of these tools in supporting realistic UX design workflows. In response, we designed and developed Flowy, an app that augments designers' information foraging process in ideation by supplementing specific user flow examples with distilled design pattern knowledge. Flowy utilizes large multimodal AI models and a high-quality user flow dataset to help designers identify and understand relevant abstract design patterns in the design space for multi-screen user flows. Our user study with professional UX designers demonstrates how Flowy supports realistic UX tasks. Our design considerations in Flowy, such as representations with appropriate levels of abstraction and assisted navigation through the solution space, are generalizable to other creative tasks and embody a human-centered, intelligence augmentation approach to using AI in UX design.

Flowy: Supporting UX Design Decisions Through AI-Driven Pattern Annotation in Multi-Screen User Flows

TL;DR

Flowy is an app that augments designers' information foraging process in ideation by supplementing specific user flow examples with distilled design pattern knowledge, and embodies a human-centered, intelligence augmentation approach to using AI in UX design.

Abstract

Many recent AI-powered UX design tools focus on generating individual static UI screens from natural language. However, they overlook the crucial aspect of interactions and user experiences across multiple screens. Through formative studies with UX professionals, we identified limitations of these tools in supporting realistic UX design workflows. In response, we designed and developed Flowy, an app that augments designers' information foraging process in ideation by supplementing specific user flow examples with distilled design pattern knowledge. Flowy utilizes large multimodal AI models and a high-quality user flow dataset to help designers identify and understand relevant abstract design patterns in the design space for multi-screen user flows. Our user study with professional UX designers demonstrates how Flowy supports realistic UX tasks. Our design considerations in Flowy, such as representations with appropriate levels of abstraction and assisted navigation through the solution space, are generalizable to other creative tasks and embody a human-centered, intelligence augmentation approach to using AI in UX design.
Paper Structure (35 sections, 3 figures, 2 tables)

This paper contains 35 sections, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Data-Driven UI/UX Design Tools
  4. Ideation with Generative AI
  5. Supporting the Information Foraging Process
  6. Formative Studies
  7. Analysis Findings
  8. Finding 1: Existing prompt-to-UI tools provide limited utility for UX design
  9. Finding 2: User flow management is highly adaptive and project-specific
  10. Finding 3: UX designers use design patterns to understand the design space and scaffold decision-making
  11. Design Goals
  12. Flowy Design & Implementation
  13. Design Pattern Annotation Pipeline
  14. Visually-Anchored Annotation Through Set-of-Mark Prompting
  15. High-Quality Design Pattern Information With Retrieval Augmented Generation
  16. ...and 20 more sections

Figures (3)

  • Figure 1: Flowy's backend design pattern annotation pipeline. The pipeline utilizes Set-of-Mark prompting (1) and Retrival-Augmented-Generation (2). It takes original screenshots of user flow examples (1), which are passed through a Segment Anything Model (SAM) and Optical Character Recognition (OCR) to identify relevant UI elements. Relevant design pattern analysis articles are retrieved from our curated knowledge base, stored in a vector database (2). The screenshots and retrieved articles are input to a multimodal AI model to generate design pattern annotations, including the pattern name, purpose, design considerations, etc. (3). Prompts are simplified to improve readability yet still highlight the core utilities.
  • Figure 2: Flowy's frontend design offers a multi-layered, flexible way to explore design examples and associated design patterns at different granularity. The homepage (A) showcases user flows from existing products. Selecting a flow opens an interactive annotated view (B), with key elements highlighted and briefly explained on hover. Clicking an annotation reveals a dedicated page (C1) with detailed pattern analysis and related design screenshot examples for side-by-side comparison (C2).
  • Figure 3: Questionnaire results from our user study.