Do MLLMs Capture How Interfaces Guide User Behavior? A Benchmark for Multimodal UI/UX Design Understanding

TL;DR

This work introduces WiserUI-Bench, a behavior-focused benchmark built from 300 real-world A/B UI variants and 684 expert interpretations to evaluate how multimodal models reason about UI/UX design and user behavior. It defines two tasks—selecting the more effective UI and post-hoc interpretation aligned with expert interpretations—to assess both predictive and explanatory capabilities. Across a broad set of proprietary and open-source MLLMs, results reveal that current models largely exhibit shallow visual-behavior reasoning, with strong position bias and limited ability to explain design-driven actions. The findings underscore the need for behavior-grounded, large-scale UI/UX data and more sophisticated multimodal reasoning to advance practical design understanding in AI systems.

Abstract

User interface (UI) design goes beyond visuals to shape user experience (UX), underscoring the shift toward UI/UX as a unified concept. While recent studies have explored UI evaluation using Multimodal Large Language Models (MLLMs), they largely focus on surface-level features, overlooking how design choices influence user behavior at scale. To fill this gap, we introduce WiserUI-Bench, a novel benchmark for multimodal understanding of how UI/UX design affects user behavior, built on 300 real-world UI image pairs from industry A/B tests, with empirically validated winners that induced more user actions. For future design progress in practice, post-hoc understanding of why such winners succeed with mass users is also required; we support this via expert-curated key interpretations for each instance. Experiments across multiple MLLMs on WiserUI-Bench for two main tasks, (1) predicting the more effective UI image between an A/B-tested pair, and (2) explaining it post-hoc in alignment with expert interpretations, show that models exhibit limited understanding of the behavioral impact of UI/UX design. We believe our work will foster research on leveraging MLLMs for visual design in user behavior contexts.

Figures (24)

• Figure 1: A real-world, behavior-aware design decision example from WiserUI-Bench, grounded in A/B test results, illustrating how UI changes steer user actions.
• Figure 2: Overview of WiserUI-Bench and two main tasks. Each instance contains a UI image pair with verified A/B test winner and expert-curated key interpretations explaining it. These span the three cognitive UX dimensions; the example includes all three, though most cover fewer. MLLMs are evaluated on (1) selecting the more effective UI/UX design by predicting the verified winner, and (2) explaining the effectiveness of a given winner, measured by whether the model captures each expert interpretation.
• Figure 3: Distribution of (a) UI change element types and (b) expert interpretations for WiserUI-Bench.
• Figure 4: Benchmark construction pipeline. (a) Raw A/B test data were collected from reliable platforms, (b) then UI images were refined for clarity. (c) Finally, key interpretations explaining the test outcomes were curated by UI/UX experts.
• Figure 5: Results of the UI/UX design interpretation task by each UX dimension on WiserUI-Bench.