Mapping the Design Space of User Experience for Computer Use Agents

TL;DR

This two-phase formative study maps the design space of user experience for LLM-powered computer use agents. Phase 1 builds a four-area UX taxonomy (User Query, Explainability, User Control, Mental Model) with 21 subcategories from nine announced agents and practitioner input. Phase 2 validates the taxonomy through a Wizard-of-Oz study with 20 participants, confirming all subcategories and yielding rich insights into end-user needs across exploratory versus specific goals, ambiguity handling, plan previews, runtime explanations, control strategies, error recovery, and privacy concerns. The findings offer a practical, non-prescriptive framework that designers can use to balance autonomy, transparency, and user involvement while adapting to diverse contexts and risk levels. Overall, the work provides a theoretically grounded map and empirical evidence to guide UX decisions in the evolving space of computer use agents.

Abstract

Large language model (LLM)-based computer use agents execute user commands by interacting with available UI elements, but little is known about how users want to interact with these agents or what design factors matter for their user experience (UX). We conducted a two-phase study to map the UX design space for computer use agents. In Phase 1, we reviewed existing systems to develop a taxonomy of UX considerations, then refined it through interviews with eight UX and AI practitioners. The resulting taxonomy included categories such as user prompts, explainability, user control, and users' mental models, with corresponding subcategories and example design features. In Phase 2, we ran a Wizard-of-Oz study with 20 participants, where a researcher acted as a web-based computer use agent and probed user reactions during normal, error-prone and risky execution. We used the findings to validate the taxonomy from Phase 1 and deepen our understand of the design space by identifying the connections between design areas and divergence in user needs and scenarios. Our taxonomy and empirical insights provide a map for developers to consider different aspects of user experience in computer use agent design and to situate their designs within users' diverse needs and scenarios.

Figures (1)

• Figure 1: The Wizard-of-Oz study setup on a mockup example website.