AgentBuilder: Exploring Scaffolds for Prototyping User Experiences of Interface Agents

TL;DR

This work addresses the challenge of prototyping user experiences for AI-powered interface agents by developing a no-code prototyping scaffold, AgentBuilder, and validating it through formative and in situ studies. A requirements elicitation study with 12 participants yields five prototyping activities and six capabilities, which are instantiated in AgentBuilder to support design, execution, and debugging of agent experiences. An in situ study with 14 participants demonstrates that non-experts can rapidly prototype diverse agent experiences and surfaces six key developer needs, including no-code interfaces, constrained task spaces, UI-definition capabilities, interaction components, execution environments, and runtime debugging tools. The findings culminate in design recommendations to democratize agent prototyping and inform the next generation of tools that bridge design, ML, and engineering for safer, more usable agent experiences.

Abstract

Interface agents powered by generative AI models (referred to as "agents") can automate actions based on user commands. An important aspect of developing agents is their user experience (i.e., agent experience). There is a growing need to provide scaffolds for a broader set of individuals beyond AI engineers to prototype agent experiences, since they can contribute valuable perspectives to designing agent experiences. In this work, we explore the affordances agent prototyping systems should offer by conducting a requirements elicitation study with 12 participants with varying experience with agents. We identify key activities in agent experience prototyping and the desired capabilities of agent prototyping systems. We instantiate those capabilities in the AgentBuilder design probe for agent prototyping. We conduct an in situ agent prototyping study with 14 participants using AgentBuilder to validate the design requirements and elicit insights on how developers prototype agents and what their needs are in this process.

Figures (10)

• Figure 1: An overview of the study. We first perform a requirements elicitation study (Section \ref{['sec:formative-study']}) to elicit the design requirements of agent prototyping systems, which consist of the five Activities involved in agent experience prototyping and six Desired Capabilities of agent prototyping systems. Based on the Activities and Desired Capabilities, we implement these design requirements in AgentBuilder (Section \ref{['sec:tool-design-probe']}) use the tool as a probe to understand agent prototyping by conducting an in situ agent prototyping study (Section \ref{['sec:in-situ-study']}) to identify developer needs in agent prototyping.
• Figure 2: An overview of the AgentBuilderprototyping interface in the Workflow tab. Developers define agent workflows (middle) using nodes and edges. Nodes (A - F) are selected from a library of actions the agent can take 1. Edges define conditions and can be selected from a library or manually written in by the user G. Developers can further specify how the agent requests user input (A, B), and what information to display when it performs UI actions (D).
• Figure 3: The AgentBuilder Prompt panel as part of the prototyping interface allows users to develop a structured prompt. Under the Edit tab 1, users can manually or automatically generate a description of their workflow A to include in their prompt. In addition, the user can input information about the agent's scope B, what knowledge it knows about the user C, and other specific instructions they want the agent to follow D. To view the assembled system prompt, the users can press the Preview tab 2.
• Figure 4: Overview of the AgentBuilder execution interface, an extension of the CowPilot agent huq-etal-2025-cowpilot. The execution environment allows users to view the UI action preview of the agent (if the developer toggles it on) in the webpage 1 and the information it conveys about its actions in the chat 4. The agent can also request user input 2, 3, 5. There are also controls to pause 6, resume, and cancel 7 the agent's execution.
• Figure 5: Overview of the AgentBuilder execution interface's Debug Mode, which can be activated (middle, right) or deactivated (left) using the Debug button 1. During this mode, users can see the tool call and agent reasoning corresponding to each message in the chat 1. Users can also enter a Detailed Debugging interface that includes the agent's inputs (i.e., webpage screenshot, webpage accessibility tree, text inputs) 5 and outputs (i.e., tool call, agent reasoning) 4. Users can navigate through the agent's previous actions using the slider 3.