Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards LLM-Based Usability Analysis for Recommender User Interfaces

Sebastian Lubos, Alexander Felfernig, Damian Garber, Viet-Man Le, Thi Ngoc Trang Tran

TL;DR

Usability assessment in recommender interfaces remains resource-intensive despite advances in algorithmic quality. The authors propose a multimodal LLM-based framework (using Gemini-2.5-flash) to analyze UI screenshots from ten public recommender platforms under two usage scenarios, guided by 11 predefined usability criteria and system/user prompts. Their evaluation yields 150 assessments in about 216 seconds, revealing that general UI aspects are well-supported, while recommender-specific explanations and user-control mechanisms are less consistently fulfilled. The work demonstrates potential for low-effort, scalable usability analysis to inform early-stage design and underscores the need for further validation against expert judgments and enhancements to capture dynamic UI behavior.

Abstract

Usability is a key factor in the effectiveness of recommender systems. However, the analysis of user interfaces is a time-consuming process that requires expertise. Recent advances in multimodal large language models (LLMs) offer promising opportunities to automate such evaluations. In this work, we explore the potential of multimodal LLMs to assess the usability of recommender system interfaces by considering a variety of publicly available systems as examples. We take user interface screenshots from multiple of these recommender platforms to cover both preference elicitation and recommendation presentation scenarios. An LLM is instructed to analyze these interfaces with regard to different usability criteria and provide explanatory feedback. Our evaluation demonstrates how LLMs can support heuristic-style usability assessments at scale to support the improvement of user experience.

Towards LLM-Based Usability Analysis for Recommender User Interfaces

TL;DR

Usability assessment in recommender interfaces remains resource-intensive despite advances in algorithmic quality. The authors propose a multimodal LLM-based framework (using Gemini-2.5-flash) to analyze UI screenshots from ten public recommender platforms under two usage scenarios, guided by 11 predefined usability criteria and system/user prompts. Their evaluation yields 150 assessments in about 216 seconds, revealing that general UI aspects are well-supported, while recommender-specific explanations and user-control mechanisms are less consistently fulfilled. The work demonstrates potential for low-effort, scalable usability analysis to inform early-stage design and underscores the need for further validation against expert judgments and enhancements to capture dynamic UI behavior.

Abstract

Usability is a key factor in the effectiveness of recommender systems. However, the analysis of user interfaces is a time-consuming process that requires expertise. Recent advances in multimodal large language models (LLMs) offer promising opportunities to automate such evaluations. In this work, we explore the potential of multimodal LLMs to assess the usability of recommender system interfaces by considering a variety of publicly available systems as examples. We take user interface screenshots from multiple of these recommender platforms to cover both preference elicitation and recommendation presentation scenarios. An LLM is instructed to analyze these interfaces with regard to different usability criteria and provide explanatory feedback. Our evaluation demonstrates how LLMs can support heuristic-style usability assessments at scale to support the improvement of user experience.

Paper Structure

This paper contains 12 sections, 7 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Usability Analysis of Recommender Interfaces
  3. Recommender Scenarios
  4. Usability Criteria
  5. LLM-based Usability Analysis
  6. Results
  7. Discussion
  8. Prioritization of Issues.
  9. Prompting Design and Context.
  10. Dynamic UI Behavior.
  11. Validation Against Expert Assessments.
  12. Conclusions

Figures (7)

  • Figure 1: System prompt for usability analysis.
  • Figure 2: User prompt template for usability analysis. Variables are highlighted in braces ({}).
  • Figure 3: Fulfillment rates for each usability criterion across all platforms. The horizontal bars indicate the proportion of platforms for which each criterion was considered fulfilled by the LLM-based analysis.
  • Figure 4: Fulfillment rates of usability criteria per platform and usage scenario.
  • Figure 5: Example usability analysis result for Goodreads on the unfulfilled information per item (G3) criterion, with corresponding explanations and improvement suggestions. The sketch illustrates the analyzed UI.
  • ...and 2 more figures