Will AI also replace inspectors? Investigating the potential of generative AIs in usability inspection

TL;DR

The paper investigates whether generative AIs can replace human inspectors in usability evaluation. By testing two state-of-the-art GenAIs (GPT-4o and Gemini 2.5 Flash) against four expert inspectors on a PEWS prototype, it quantifies precision, recall, and F1 and analyzes defect overlap and duplication. Findings show that inspectors achieve higher precision and broader defect coverage, while AIs are faster but yield more false positives and duplicates; a hybrid approach offers the best balance of recall and precision. The work concludes that GenAIs are valuable augmentation tools for usability inspection, with practical implications for scalable QA, early-design reviews, and future research into more robust AI-assisted evaluation pipelines.

Abstract

Usability inspection is a well-established technique for identifying interaction issues in software interfaces, thereby contributing to improved product quality. However, it is a costly process that requires time and specialized knowledge from inspectors. With advances in Artificial Intelligence (AI), new opportunities have emerged to support this task, particularly through generative models capable of interpreting interfaces and performing inspections more efficiently. This study examines the performance of generative AIs in identifying usability problems, comparing them to those of experienced human inspectors. A software prototype was evaluated by four specialists and two AI models (GPT-4o and Gemini 2.5 Flash), using metrics such as precision, recall, and F1-score. While inspectors achieved the highest levels of precision and overall coverage, the AIs demonstrated high individual performance and discovered many novel defects, but with a higher rate of false positives and redundant reports. The combination of AIs and human inspectors produced the best results, revealing their complementarity. These findings suggest that AI, in its current stage, cannot replace human inspectors but can serve as a valuable augmentation tool to improve efficiency and expand defect coverage. The results provide evidence based on quantitative analysis to inform the discussion on the role of AI in usability inspections, pointing to viable paths for its complementary use in software quality assessment contexts.

Figures (2)

• Figure 1: Research methodology.
• Figure 2: Patient's clinical assessment form.