MLLM as a UI Judge: Benchmarking Multimodal LLMs for Predicting Human Perception of User Interfaces

TL;DR

This work investigates whether multimodal LLMs can serve as early evaluators of user interfaces by benchmarking three state-of-the-art models against human judgments across 30 UIs and nine UI factors. It conducts two tasks—absolute score prediction and pairwise preference prediction—to assess alignment, using human data collected via crowdsourcing as a ground truth. The results show moderate alignment for some cognitive and perceptual factors, with stronger performance in pairwise rankings when UI differences are large, but limited precision for fine-grained, emotion-related aspects. The authors provide a benchmark dataset to enable fine-tuning and RLHF-based improvements, highlighting that MLLMs should supplement but not replace human UX research, especially in low-stakes, rapid-design contexts.

Abstract

In an ideal design pipeline, user interface (UI) design is intertwined with user research to validate decisions, yet studies are often resource-constrained during early exploration. Recent advances in multimodal large language models (MLLMs) offer a promising opportunity to act as early evaluators, helping designers narrow options before formal testing. Unlike prior work that emphasizes user behavior in narrow domains such as e-commerce with metrics like clicks or conversions, we focus on subjective user evaluations across varied interfaces. We investigate whether MLLMs can mimic human preferences when evaluating individual UIs and comparing them. Using data from a crowdsourcing platform, we benchmark GPT-4o, Claude, and Llama across 30 interfaces and examine alignment with human judgments on multiple UI factors. Our results show that MLLMs approximate human preferences on some dimensions but diverge on others, underscoring both their potential and limitations in supplementing early UX research.

Figures (8)

• Figure 1: User interfaces split by domains: An overview of a sample of the UIs evaluated by humans and MLLMs can be divided into three domains: landing pages, digital receipts, and catalogs. Here we present low-fidelity versions of the screens that we presented to the users. The users saw high-fidelity, branded, and in-product versions of these screens that have been anonymized.
• Figure 2: Example survey question shown to users. Consists of instructions on top, the UI on the left, and the nine factor, Likert scale questions on the right.
• Figure 3: Accuracy in predicting pairwise human preferences over UIs as a function of the absolute difference of their average human scores. A higher accuracy generally correlates with a larger difference in the human scores. The score-difference bin values were calculated as the difference between the average human ratings of the two UIs. This number is used to infer the difficulty of the question: smaller values indicate more difficult questions, while larger values imply easier ones.
• Figure 4: Results when pairwise data is created for MLLMs and humans in the same way. For both human and MLLM evaluation sets, we take the absolute value gathered in task 1, compare the two absolute values, and take the higher of the two. Then we compare the human results to the MLLM results to get the agreement score presented here.
• Figure 5: Mean Absolute Error (lower is better): This result shows that the mean absolute error for all models consistently is less than 1, with "interesting" being the exception. These results show that the models can reliably score within one point of the human scores when given a Likert scale UI question.