How do Visual Attributes Influence Web Agents? A Comprehensive Evaluation of User Interface Design Factors

TL;DR

We address how visual webpage attributes shape decision-making in vision-language model (VLM) web agents by introducing Visual Attribute Factors (VAF), a three-stage evaluation pipeline (variant generation, human-like browsing, and validation). Using 8 variant families (48 variants) across 5 real-world sites and 4 agents, VAF quantifies how visual cues influence Target Click Rate (TCR) and Target Mention Rate (TMR). Results show background color contrast, item size, position, and card clarity strongly affect agent actions, while font styling, text color, and image clarity have smaller effects; agents display layout priors akin to humans but are brittle when visual grounding is degraded. These insights support designing more robust, human-aligned web interfaces for AI agents and highlight differences between human and agent perception in dynamic online environments.

Abstract

Web agents have demonstrated strong performance on a wide range of web-based tasks. However, existing research on the effect of environmental variation has mostly focused on robustness to adversarial attacks, with less attention to agents' preferences in benign scenarios. Although early studies have examined how textual attributes influence agent behavior, a systematic understanding of how visual attributes shape agent decision-making remains limited. To address this, we introduce VAF, a controlled evaluation pipeline for quantifying how webpage Visual Attribute Factors influence web-agent decision-making. Specifically, VAF consists of three stages: (i) variant generation, which ensures the variants share identical semantics as the original item while only differ in visual attributes; (ii) browsing interaction, where agents navigate the page via scrolling and clicking the interested item, mirroring how human users browse online; (iii) validating through both click action and reasoning from agents, which we use the Target Click Rate and Target Mention Rate to jointly evaluate the effect of visual attributes. By quantitatively measuring the decision-making difference between the original and variant, we identify which visual attributes influence agents' behavior most. Extensive experiments, across 8 variant families (48 variants total), 5 real-world websites (including shopping, travel, and news browsing), and 4 representative web agents, show that background color contrast, item size, position, and card clarity have a strong influence on agents' actions, whereas font styling, text color, and item image clarity exhibit minor effects.

Figures (3)

• Figure 1: Overview of VAF. Our evaluation pipeline consists of three stages: (i) variant generation, where we construct content-preserving visual variants by modifying the CSS of a designated target item on real-world webpages; (ii) human-like browsing interaction, where agents navigate the page via scrolling and clicking, resembling human browsing behavior; (iii) validating through both click and reasoning from agents, which we use the Target Click Rate and Target Mention Rate to jointly evaluate the effect of visual attributes. By quantitatively comparing agent behavior on the original webpage versus its visually modified variants, we measure how visual attributes influence web-agent decision-making.
• Figure 2: Heatmap of $\Delta=\mathrm{TCR}_{\mathrm{variant}}-\mathrm{TCR}_{\mathrm{original}}$. Larger $\Delta$ indicates that the variant gets more clicks than the original target item. Across diverse models and scenarios, we observe that (1) high background color contrast and enlarged item card consistently increase attraction; (2) item position strongly affects decisions, with agents biased toward selecting the first few items; (3) item image clarity has limited impact, whereas entire card clarity has a stronger effect on agent actions; and (4) font style and text color variants have a relatively minor influence on decision-making in general. nan indicates variants are not applicable in the corresponding scenario.
• Figure 3: A qualitative example of click distribution comparison on the Booking. The target item (i.e., first item on the original webpage) is marked with a blue frame. After variant generation, the target click rate of UI-TARS 7B decreases from 18% to 0% on average across 50 trials, suggesting that the agent tends to click items near the top of the page.