Prompting for Discovery: Flexible Sense-Making for AI Art-Making with Dreamsheets

TL;DR

DreamSheets tackles the challenge of sense-making in large, opaque Text-to-Image design spaces by pairing a spreadsheet-based interface with LLM-powered prompt manipulation and rapid visual feedback. Through a novice lab study and an extended two-week expert study, the authors reveal strategies for iterative prompt exploration, parametric and semantic axes, and multidimensional workbooks that artists can reuse and adapt. The work contributes a detailed DreamSheets design, an implementation that leverages Google Sheets with a caching backend, and a 2.0 UI mockup co-designed with expert artists to guide future interfaces. These findings demonstrate how flexible, reusable scaffolds can support diverse creative workflows and broaden access to sophisticated TTI exploration beyond goal-driven prompts. The study further argues for treating exploration interfaces as sensemaking tools that empower users to understand and navigate the input-output mappings of generative models, not merely to hit specific outputs.

Abstract

Design space exploration (DSE) for Text-to-Image (TTI) models entails navigating a vast, opaque space of possible image outputs, through a commensurately vast input space of hyperparameters and prompt text. Minor adjustments to prompt input can surface unexpectedly disparate images. How can interfaces support end-users in reliably steering prompt-space explorations towards interesting results? Our design probe, DreamSheets, supports exploration strategies with LLM-based functions for assisted prompt construction and simultaneous display of generated results, hosted in a spreadsheet interface. The flexible layout and novel generative functions enable experimentation with user-defined workflows. Two studies, a preliminary lab study and a longitudinal study with five expert artists, revealed a set of strategies participants use to tackle the challenges of TTI design space exploration, and the interface features required to support them - like using text-generation to define local "axes" of exploration. We distill these insights into a UI mockup to guide future interfaces.

Figures (13)

• Figure 1: The DreamSheets implementation. LLM (A) and TTI (B) functions fetch from separate endpoints of the DreamSheets cloud server (C) which forwards requests to the OpenAI ChatGPT and Stability.ai Stable Diffusion cloud-based APIs. TTI requests are cached (D) using a hash of (prompt text, seed, classifier-free guidance) as a key.
• Figure 2: Examples of prompts authored by participants in the first study
• Figure 3: Number of times participants used each of the LLM-based functions (listed in order of ascending frequency) during the preliminary lab study activity. There were 4,737 calls to the TTI() image generation function.
• Figure 4: Above, LLM-based function use across the 2-week expert use study. Individual colors represent individual expert participants consistently across functions (5 total). Below, the number of times each expert participant made a unique call to the image generation and text generation functions during the 2-week study.
• Figure 5: We abstract the Prompt-Input and Visual-Output space as two dimensions that the generative model maps between, and use these abstractions to illustrate the space(s) that TTI users are targeting via different exploration strategies. Steering towards desirable results with iterative prompt refinement is a common strategy, but sparsely presented results can confound productive sense-making. DreamSheets users prototyped iterative sheet-systems that leveraged the "infinite-canvas" and rich history-keeping affordances of digital spreadsheets to conduct large-scale comparison across results, and to arrange and repurpose past explorations. With a structured, scalable results display, users can stumble upon interesting outliers without being confounded by them.