Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

UXAgent: An LLM Agent-Based Usability Testing Framework for Web Design

Yuxuan Lu, Bingsheng Yao, Hansu Gu, Jing Huang, Jessie Wang, Yang Li, Jiri Gesi, Qi He, Toby Jia-Jun Li, Dakuo Wang

TL;DR

The paper tackles the challenge of inflexible usability testing and participant recruitment by introducing UXAgent, an LLM-Agent-based framework for simulating usability studies. It combines a Persona Generator, Universal Browser Connector, a two-loop LLM Agent with a Memory Stream, and a chat interface to generate thousands of synthetic participants and multimodal data. The authors validate the approach with a user study of five UX researchers analyzing 60 simulated sessions on shopping tasks across WebArena and Google Flights, yielding insights on data trust, interpretation, biases, and realism. The work demonstrates potential to accelerate pilot testing and reduce risks for real participants while highlighting ethical and privacy considerations and the need for caution about realism and biases.

Abstract

Usability testing is a fundamental yet challenging (e.g., inflexible to iterate the study design flaws and hard to recruit study participants) research method for user experience (UX) researchers to evaluate a web design. Recent advances in Large Language Model-simulated Agent (LLM-Agent) research inspired us to design UXAgent to support UX researchers in evaluating and reiterating their usability testing study design before they conduct the real human subject study. Our system features an LLM-Agent module and a universal browser connector module so that UX researchers can automatically generate thousands of simulated users to test the target website. The results are shown in qualitative (e.g., interviewing how an agent thinks ), quantitative (e.g., # of actions), and video recording formats for UX researchers to analyze. Through a heuristic user evaluation with five UX researchers, participants praised the innovation of our system but also expressed concerns about the future of LLM Agent-assisted UX study.

UXAgent: An LLM Agent-Based Usability Testing Framework for Web Design

TL;DR

The paper tackles the challenge of inflexible usability testing and participant recruitment by introducing UXAgent, an LLM-Agent-based framework for simulating usability studies. It combines a Persona Generator, Universal Browser Connector, a two-loop LLM Agent with a Memory Stream, and a chat interface to generate thousands of synthetic participants and multimodal data. The authors validate the approach with a user study of five UX researchers analyzing 60 simulated sessions on shopping tasks across WebArena and Google Flights, yielding insights on data trust, interpretation, biases, and realism. The work demonstrates potential to accelerate pilot testing and reduce risks for real participants while highlighting ethical and privacy considerations and the need for caution about realism and biases.

Abstract

Usability testing is a fundamental yet challenging (e.g., inflexible to iterate the study design flaws and hard to recruit study participants) research method for user experience (UX) researchers to evaluate a web design. Recent advances in Large Language Model-simulated Agent (LLM-Agent) research inspired us to design UXAgent to support UX researchers in evaluating and reiterating their usability testing study design before they conduct the real human subject study. Our system features an LLM-Agent module and a universal browser connector module so that UX researchers can automatically generate thousands of simulated users to test the target website. The results are shown in qualitative (e.g., interviewing how an agent thinks ), quantitative (e.g., # of actions), and video recording formats for UX researchers to analyze. Through a heuristic user evaluation with five UX researchers, participants praised the innovation of our system but also expressed concerns about the future of LLM Agent-assisted UX study.

Paper Structure

This paper contains 35 sections, 6 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Challenges in Usability Testing
  4. LLM Agent and LLM Web Agent
  5. UXAgent: LLM-Agent-Based System for Usability Testing
  6. Persona Generator Module Design
  7. LLM Agent Designed for Web Environment Interactions
  8. Universal Browser Connector Module
  9. Chat Interface
  10. Simulation Environment and Task for LLM Agents
  11. User Study
  12. Participant Recruitment and Participant Task
  13. Study Findings
  14. How Did UX Researchers Use Our UXAgent System?
  15. Gaining trust of data
  16. ...and 20 more sections

Figures (6)

  • Figure 1: User study session and an interactive chat interface with LLM Agent
  • Figure 2: Participant's perception towards using UXAgent in User Study
  • Figure 3: Example of a part of a web page processed by the Univerasl Web Connector and the Perception Module. The Univerasl Web Connector parses the raw web page (left) into a simplified HTML structure (right), and the Perception Module generates a descriptive observation memory (bottom) for each segment.
  • Figure 4: HTML Parser Example of multiple nested elements.
  • Figure 5: The structure of the LLM Agent's Reasoning Architecture
  • ...and 1 more figures