Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Synthetic Cognitive Walkthrough: Aligning Large Language Model Performance with Human Cognitive Walkthrough

Ruican Zhong, David W. McDonald, Gary Hsieh

TL;DR

The paper examines whether off-the-shelf LLMs (GPT-4 and Gemini-2.5-pro) can simulate human cognitive walkthroughs (CW) to evaluate UI learnability on two mobile apps. It shows LLMs achieve higher task completion and more optimal navigation than humans but identify far fewer potential failure points, highlighting fundamental differences in CW behavior. A follow-up study demonstrates that with-context prompting, LLMs more consistently predict human-identified failure points, suggesting a practical pathway to scale CW while preserving interpretability. The work concludes that LLMs can complement traditional usability testing by providing scalable UI insights, provided designers account for behavioral differences and leverage context-rich prompts.

Abstract

Conducting usability testing like cognitive walkthrough (CW) can be costly. Recent developments in large language models (LLMs), with visual reasoning and UI navigation capabilities, present opportunities to automate CW. We explored whether LLMs (GPT-4 and Gemini-2.5-pro) can simulate human behavior in CW by comparing their walkthroughs with human participants. While LLMs could navigate interfaces and provide reasonable rationales, their behavior differed from humans. LLM-prompted CW achieved higher task completion rates than humans and followed more optimal navigation paths, while identifying fewer potential failure points. However, follow-up studies demonstrated that with additional prompting, LLMs can predict human-identified failure points, aligning their performance with human participants. Our work highlights that while LLMs may not replicate human behaviors exactly, they can be leveraged for scaling usability walkthroughs and providing UI insights, offering a valuable complement to traditional usability testing.

Synthetic Cognitive Walkthrough: Aligning Large Language Model Performance with Human Cognitive Walkthrough

TL;DR

The paper examines whether off-the-shelf LLMs (GPT-4 and Gemini-2.5-pro) can simulate human cognitive walkthroughs (CW) to evaluate UI learnability on two mobile apps. It shows LLMs achieve higher task completion and more optimal navigation than humans but identify far fewer potential failure points, highlighting fundamental differences in CW behavior. A follow-up study demonstrates that with-context prompting, LLMs more consistently predict human-identified failure points, suggesting a practical pathway to scale CW while preserving interpretability. The work concludes that LLMs can complement traditional usability testing by providing scalable UI insights, provided designers account for behavioral differences and leverage context-rich prompts.

Abstract

Conducting usability testing like cognitive walkthrough (CW) can be costly. Recent developments in large language models (LLMs), with visual reasoning and UI navigation capabilities, present opportunities to automate CW. We explored whether LLMs (GPT-4 and Gemini-2.5-pro) can simulate human behavior in CW by comparing their walkthroughs with human participants. While LLMs could navigate interfaces and provide reasonable rationales, their behavior differed from humans. LLM-prompted CW achieved higher task completion rates than humans and followed more optimal navigation paths, while identifying fewer potential failure points. However, follow-up studies demonstrated that with additional prompting, LLMs can predict human-identified failure points, aligning their performance with human participants. Our work highlights that while LLMs may not replicate human behaviors exactly, they can be leveraged for scaling usability walkthroughs and providing UI insights, offering a valuable complement to traditional usability testing.

Paper Structure

This paper contains 36 sections, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Cognitive Walkthrough and Automated User Evaluation
  4. UI Understanding, Navigation, and User Simulation
  5. Cognitive Walkthrough Prompt Design
  6. Pipeline Setup
  7. Facilitator
  8. Evaluator
  9. Pipeline Process
  10. Study 1: Comparing LLM and Human Conducted Cognitive Walkthrough
  11. Procedure
  12. Synthetic Cognitive Walkthrough (LLM Set)
  13. Human Set
  14. Measures
  15. Analyses
  16. ...and 21 more sections

Figures (3)

  • Figure 1: These two figures are example screens in the language learning app.
  • Figure 2: These two figures are example screens in the booking app.
  • Figure 3: This figure presents the study 1 results, comparing human, GPT, and Gemini runs on task completion rate, number of steps to complete a task, and JS score.