Beyond Code Generation: LLM-supported Exploration of the Program Design Space

TL;DR

This work tackles the problem that conventional LLM-assisted programming emphasizes single-point code outputs, limiting exploration of the broader program design space. It introduces Pail, an agent-based IDE that couples a chat-driven agent with a DesignPanel to surface design alternatives, rationales, and implicit AI decisions, while enabling speculative problem formulations and incremental, diff-based code updates. An 11-participant lab study shows that Pail supports broader exploration of requirements and user-centered designs, but also reveals substantial attention-management and information-overload challenges as AI-driven changes propagate. The findings emphasize the need for careful information shaping and attention control in future AI-augmented programming environments and outline design implications and directions for further research.

Abstract

In this work, we explore explicit Large Language Model (LLM)-powered support for the iterative design of computer programs. Program design, like other design activity, is characterized by navigating a space of alternative problem formulations and associated solutions in an iterative fashion. LLMs are potentially powerful tools in helping this exploration; however, by default, code-generation LLMs deliver code that represents a particular point solution. This obscures the larger space of possible alternatives, many of which might be preferable to the LLM's default interpretation and its generated code. We contribute an IDE that supports program design through generating and showing new ways to frame problems alongside alternative solutions, tracking design decisions, and identifying implicit decisions made by either the programmer or the LLM. In a user study, we find that with our IDE, users combine and parallelize design phases to explore a broader design space -- but also struggle to keep up with LLM-originated changes to code and other information overload. These findings suggest a core challenge for future IDEs that support program design through higher-level instructions given to LLM-based agents: carefully managing attention and deciding what information agents should surface to program designers and when.

Figures (3)

• Figure 1: Pail's capabilities overlaid on the UK Design Council's Double Diamond design_council_framework_2004. Building beyond a "prompt"-to-"code" view of LLM code generation (top), Pail (bottom) embeds this capability within an IDE that supports developers through a larger design process: by abstracting up from a specific problem to a higher-level one, exploring alternative problems and alternative solutions, tracking design goals and requirements, and surfacing implicit decisions (green boxes).
• Figure 2: An overview of Pail's user interface and related agents. Pail engages users in defining and refining ( ) design goals, including target audience and desired impact, while tracking ( ) confirmed requirements and ( ) decisions implicit in an LLM's synthesized code, surfacing ( ) useful abstractions, and offering explanations and alternatives across the program design space. A full-scale reproduction of the design pane's four views appears in Figure \ref{['fig:design-goals-snapshot']}.
• Figure 3: A screenshot of the contents of each of the four components of the design panel, from P9. The third question in Design Questions & Goals(a) ("How can we effectively [...]?") is expanded to show the rationale for that question along with three alternatives. The mouse hover over the third alternative ("Crowdsource [...]") reveals a speculated trade-off for trying that option. In Pail, these four sections are organized vertically in the design panel. Items with a finger are important, dotted underlines allow hover (see (a)) for trade-offs. Green background and red strikethrough indicate recent changes.