ChatGPT and U(X): A Rapid Review on Measuring the User Experience
Katie Seaborn
TL;DR
This rapid review synthesizes how researchers have quantitatively measured the user experience (UX) of ChatGPT up to December 2024, focusing on manipulated independent variables, measured dependent variables, and measurement methods across 64 studies (N=15,759). It reveals substantial heterogeneity in constructs and instruments, with limited validation and reliability checks, and notes gaps such as underexplored voice modalities and social-intelligence factors. The authors introduce two preliminary frameworks to categorize IVs and DVs and provide an open data set to promote standardization and replication in future work. Overall, the study highlights the need for theory-driven, instrument-backed, GPT-version-aware UX measurement to enable robust meta-analyses and broader generalizability across contexts.
Abstract
ChatGPT, powered by a large language model (LLM), has revolutionized everyday human-computer interaction (HCI) since its 2022 release. While now used by millions around the world, a coherent pathway for evaluating the user experience (UX) ChatGPT offers remains missing. In this rapid review (N = 58), I explored how ChatGPT UX has been approached quantitatively so far. I focused on the independent variables (IVs) manipulated, the dependent variables (DVs) measured, and the methods used for measurement. Findings reveal trends, gaps, and emerging consensus in UX assessments. This work offers a first step towards synthesizing existing approaches to measuring ChatGPT UX, urgent trajectories to advance standardization and breadth, and two preliminary frameworks aimed at guiding future research and tool development. I seek to elevate the field of ChatGPT UX by empowering researchers and practitioners in optimizing user interactions with ChatGPT and similar LLM-based systems.