Systematic Literature Review of Automation and Artificial Intelligence in Usability Issue Detection

TL;DR

In this systematic review of 155 publications, a comprehensive overview of the current state of the art for automated usability issue detection is offered, which analyzes trends, paradigms, and the technical context in which they are applied.

Abstract

Usability issues can hinder the effective use of software. Therefore, various techniques are deployed to diagnose and mitigate them. However, these techniques are costly and time-consuming, particularly in iterative design and development. A substantial body of research indicates that automation and artificial intelligence can enhance the process of obtaining usability insights. In our systematic review of 155 publications, we offer a comprehensive overview of the current state of the art for automated usability issue detection. We analyze trends, paradigms, and the technical context in which they are applied. Finally, we discuss the implications and potential directions for future research.

Figures (14)

• Figure 1: Funnel diagram of the systematic literature survey protocol, portraying the downward filter of publications to obtain unique, relevant and high-quality articles. The acquisition, selection, and extraction process resulted in 155 publications being included for analysis.
• Figure 2: Distribution of primary studies in the corpus, categorized by journal impact quartiles and conference rankings (a) and primary study publisher distribution bar chart (b). The question mark (?) represents unranked conferences. The dataset shows similar ratios of journal articles and conference papers, along with their quality indicators. The most prevalent publishers include ACM with 48 publications in total, followed by Springer Nature, IEEE, and Elsevier.
• Figure 3: Yearly distribution of research publications indicates a slight upward trend, with per-year citation counts showing a notable increase over the past three years.
• Figure 4: Distribution of research objectives in publications on automated detection of usability problems. The most common objective is Usability attribute evaluation, accounting for 34% of publications, followed by Affective state detection and Automatic guideline evaluation.
• Figure 5: Per-year distribution of research objectives involving automated usability problem detection. Research assistants have become more prevalent in recent years.