Generative and Malleable User Interfaces with Generative and Evolving Task-Driven Data Model

TL;DR

This work presents Jelly, a prototype system that uses LLMs to generate evolving task-driven data models (object-relational schemas, dependency graphs, and structured data) and translates them into UI specifications for generative, malleable user interfaces. By enabling end-user natural language prompts and direct manipulation that update the underlying model, Jelly supports continuous customization, flexible task representation, and interpretable UI generation. Technical evaluation shows that LLMs reliably produce relevant entities and dependencies, while a user study demonstrates that users can effectively organize information, customize interfaces, and pursue open-ended tasks with persistent, task-oriented spaces. The findings suggest a promising direction for personalizable, context-preserving interfaces, with future work focusing on richer dependency modeling, higher-level schema transformations, enhanced view management, external data integration, and broader personalization capabilities.

Abstract

Unlike static and rigid user interfaces, generative and malleable user interfaces offer the potential to respond to diverse users' goals and tasks. However, current approaches primarily rely on generating code, making it difficult for end-users to iteratively tailor the generated interface to their evolving needs. We propose employing task-driven data models-representing the essential information entities, relationships, and data within information tasks-as the foundation for UI generation. We leverage AI to interpret users' prompts and generate the data models that describe users' intended tasks, and by mapping the data models with UI specifications, we can create generative user interfaces. End-users can easily modify and extend the interfaces via natural language and direct manipulation, with these interactions translated into changes in the underlying model. The technical evaluation of our approach and user evaluation of the developed system demonstrate the feasibility and effectiveness of the proposed generative and malleable UIs.

Figures (7)

• Figure 1: Our proposed pipeline for generative and malleable UIs. The pipeline takes the user prompt as input and employs LLMs to generate a data model describing the task. This model serves as the foundation for generating UI specifications, which guide UI composition and state management. The specifications are then used to render the UI based on predefined rules. Users can iteratively customize both the data model and UI specifications through interactions.
• Figure 2: The format of object-relational schema and the annotation of attributes within the schema. (a) task attributes of the schema, (b) entities described in the schema; (c1) single data value attribute, (c2) array attribute, (c3) dictionary attribute, (c) pointer attribute; (d1) annotated single data value, (d2) annotated array attribute, (d3) annotated pointer attribute
• Figure 3: [0.85]Jelly consists of the Schema View, the Chat View, and the Generated Interfaces. The task object is presented as the Home Panel (1). Users can navigate to Entity Panels (2) by expanding items in other panels (a-b) or using the navigation button on the right side of the interface (c). Users may click to reveal details of an item (d) and hover over an item to trigger synchronized highlighting across panels (e). Users can customize the UIs and data through continuous prompting (f) and direct manipulation (g-h), with all actions preserved as a traceable history in the Chat View.
• Figure 4: [0.85]Jelly currently supports switching between three different representations of data for an entity pane, including (a) Map View, (b) List View (where each item is expanded to show more details, e.g., the Shopping Items attribute is directly rendered for people to directly scroll through the list), and (c) Table View (where attributes may be folded to fit in the cells, e.g., the Shopping Items list is folded into a button. The user may click on it to reveal the full list).
• Figure 5: Generating Structured Data with [0.85]Jelly. Users can create an empty object card (a), manually input some of the attributes, and optionally provide a prompt specifying their preference over other attributes (b). [0.85]Jelly then auto-completes the remaining attributes based on the provided input (c).