CrowdGenUI: Aligning LLM-Based UI Generation with Crowdsourced User Preferences

TL;DR

CrowdGenUI presents a modular framework that augments LLM-driven UI generation with crowdsourced user preferences to produce task-aware and user-aligned widgets. By integrating a crowdsourced preference library with LLM reasoning and code generation, the approach demonstrates improved alignment with user needs in an image editing prototype, validated through a user study with $N=78$ and a crowdsourcing library of $720$ preferences from $50$ participants. Key contributions include the design of a four-component workflow (user context, preference library, widget candidates, and LLM outputs), a crowdsourcing methodology for collecting UI preferences, and empirical evidence that larger preference libraries yield more consistent widget reasoning while smaller libraries offer cost-effective personalization. The work highlights the potential of crowdsourced guidance to generalize beyond the original task domain, informing scalable, adaptive, human-centered UI generation in diverse domains and laying groundwork for future extensions involving broader task categories and agentic AI collaboration.

Abstract

Large Language Models (LLMs) have demonstrated remarkable potential across various design domains, including user interface (UI) generation. However, current LLMs for UI generation tend to offer generic solutions that lack a nuanced understanding of task context and user preferences. We present CrowdGenUI, a framework that enhances LLM-based UI generation with crowdsourced user preferences. This framework addresses the limitations by guiding LLM reasoning with real user preferences, enabling the generation of UI widgets that reflect user needs and task-specific requirements. We evaluate our framework in the image editing domain by collecting a library of 720 user preferences from 50 participants, covering preferences such as predictability, efficiency, and explorability of various UI widgets. A user study (N=78) demonstrates that UIs generated with our preference-guided framework can better match user intentions compared to those generated by LLMs alone, highlighting the effectiveness of our proposed framework. We further discuss the study findings and present insights for future research on LLM-based user-centered UI generation.

Figures (17)

• Figure 1: Overview of the CrowdGenUI framework. This framework consists of four main components: it takes (1) user context as input, including task description and UI control preferences, and this information is passed to the LLM together with (2) a user preference library and (3) a collection of UI widget candidates. All the input data is used to guide the LLM reasoning process to obtain (4) generated UI widgets that are aligned with user preferences and task requirements.
• Figure 2: Example usage scenario of CrowdGenUI. A user wants to change the color of an image to fall colors. They provide the image, describe the task, and specify their UI control preference. Taking user input, CrowdGenUI reasons the proper UI widgets and generates the code for widget implementation. The user uses the generated widgets to edit the image and can further update their input to obtain updated UIs, e.g., from efficient experiments using preset options to fine-grained control using a slider.
• Figure 3: System implementation pipeline. The system starts with natural language input specifying the task description and user preferences. With this input, the LLM applies chain-of-thought (CoT) reasoning to obtain reasoned widgets. Based on this reasoning process, the crowdsourced user preference library, containing the crowdsourcing tasks, user-preferred widgets, and their rationale, is integrated to augment widget reasoning. Additionally, the Jupyter Widget Library widget candidates are fed to the LLM for reasoned widget implementation. Lastly, the reasoned widgets and reference code snippets are passed to the LLM for code generation in Python. This allows widget implementation and interaction in Jupyter Notebook.
• Figure 4: Crowdsourcing user interface for task: adjusting an image to fall colors (image_adjust_fall_color).
• Figure 5: User preference distribution of tasks 1-4 (task set 1) collected in the crowdsourcing study.