Catching UX Flaws in Code: Leveraging LLMs to Identify Usability Flaws at the Development Stage

TL;DR

The paper investigates whether GPT-4o can perform Nielsen-based usability heuristics on web interfaces during development, applying the ten heuristics to 30 open-source sites with three independent sessions per site to assess reliability of automated UX evaluations. The study finds moderate consistency for detecting issue presence (Cohen's Kappa ~0.5; exact agreement ~84%) but greater variability in severity ratings (weighted kappa ~0.63; exact ~56%; Krippendorff's Alpha near zero), highlighting both the promise and limitations of automated, early-stage UX testing. It demonstrates a feasible, reproducible pipeline for AI-assisted usability evaluation that can support resource-constrained teams, while underscoring the need for human oversight and further benchmarking against expert evaluations. The work lays groundwork for integrating LLM-driven UX checks into development workflows and points to future directions for improving prompt design and reliability across scenarios.

Abstract

Usability evaluations are essential for ensuring that modern interfaces meet user needs, yet traditional heuristic evaluations by human experts can be time-consuming and subjective, especially early in development. This paper investigates whether large language models (LLMs) can provide reliable and consistent heuristic assessments at the development stage. By applying Jakob Nielsen's ten usability heuristics to thirty open-source websites, we generated over 850 heuristic evaluations in three independent evaluations per site using a pipeline of OpenAI's GPT-4o. For issue detection, the model demonstrated moderate consistency, with an average pairwise Cohen's Kappa of 0.50 and an exact agreement of 84%. Severity judgments showed more variability: weighted Cohen's Kappa averaged 0.63, but exact agreement was just 56%, and Krippendorff's Alpha was near zero. These results suggest that while GPT-4o can produce internally consistent evaluations, especially for identifying the presence of usability issues, its ability to judge severity varies and requires human oversight in practice. Our findings highlight the feasibility and limitations of using LLMs for early-stage, automated usability testing, and offer a foundation for improving consistency in automated User Experience (UX) evaluation. To the best of our knowledge, our work provides one of the first quantitative inter-rater reliability analyses of automated heuristic evaluation and highlights methods for improving model consistency.

Figures (2)

• Figure 1: Distribution of issues found across all evaluations. A large majority of heuristics were flagged as having usability issues by the model.
• Figure 2: Distribution of weighted Cohen's Kappa values per site (eval1 vs eval2) for severity ratings.