UI Remix: Supporting UI Design Through Interactive Example Retrieval and Remixing

TL;DR

UI Remix addresses the challenge that end-user designers face in articulating UI intent and trusting AI-guided designs by combining a three-mode interaction (Chat, Search, Apply) with global and local remix of real-world UI examples. Powered by a multimodal retrieval-augmented generation model, it surfaces traceable examples with source metadata to support reasoning and trust, while live previews and code diff-patch updates keep design work iterative and controllable. In a 24-participant within-subject study, UI Remix improved exploration, goal-oriented iteration, and willingness to adopt the system, though final output quality on short tasks did not significantly exceed a baseline. The work demonstrates how example-driven, transparent AI collaboration can empower end users to design with greater creativity and confidence, and outlines future directions for extending to multi-screen flows and proactive guidance.

Abstract

Designing user interfaces (UIs) is a critical step when launching products, building portfolios, or personalizing projects, yet end users without design expertise often struggle to articulate their intent and to trust design choices. Existing example-based tools either promote broad exploration, which can cause overwhelm and design drift, or require adapting a single example, risking design fixation. We present UI Remix, an interactive system that supports mobile UI design through an example-driven design workflow. Powered by a multimodal retrieval-augmented generation (MMRAG) model, UI Remix enables iterative search, selection, and adaptation of examples at both the global (whole interface) and local (component) level. To foster trust, it presents source transparency cues such as ratings, download counts, and developer information. In an empirical study with 24 end users, UI Remix significantly improved participants' ability to achieve their design goals, facilitated effective iteration, and encouraged exploration of alternative designs. Participants also reported that source transparency cues enhanced their confidence in adapting examples. Our findings suggest new directions for AI-assisted, example-driven systems that empower end users to design with greater control, trust, and openness to exploration.

Figures (2)

• Figure 1: MMRAG Pipeline. The process begins with preprocessing UI screenshots from the Mobbin mobbin, Interaction Mining interactionminingODIM, and MobileViews gao2024mobileviews datasets, along with metadata from the Google Play Store. Each screenshot is linked to its corresponding metadata through indexing. Next, the UI screenshots are passed through an encoder to generate embeddings. The embeddings, together with the corresponding image content, are stored in ChromaDB. When a user submits a query, it is converted into embeddings, and the system retrieves and ranks relevant screenshots using cosine similarity. The retrieved UIs, along with their metadata, are then displayed in the Example Gallery for the user to browse and select desirable designs. Once a user selects a UI, the selected design --- combined with the user's query, current UI code, and target area (used only for local remix) --- is formatted into a complete prompt and sent to the MLLM, which generates updated UI code in a diff-patch format. The system backend parses these responses using the unidiff and difflib packages, updates the UI code, and renders them as UI previews.
• Figure 2: A screenshot of GPT-Canvas baseline system.