The Behavioral Fabric of LLM-Powered GUI Agents: Human Values and Interaction Outcomes

TL;DR

The paper investigates how explicit human values and preferences shape the reasoning and action trajectories of LLM-powered web GUI agents, using a controlled, multi-domain testbed with 14 tasks and four agent models. It demonstrates that value and preference prompts systematically steer behavior and that environmental cues, such as promotional UI, can strongly override value-driven deliberation, leading to a value–action gap. The study adds an open-source, value-sensitive testbed and an empirical analysis of how user values influence agent reasoning, outcomes, and generic defaults in the absence of guidance. The findings highlight the need for value-aware evaluation, transparent reasoning traces, and scalable oversight to build aligned, user-controllable AI agents in web environments.

Abstract

Large Language Model (LLM)-powered web GUI agents are increasingly automating everyday online tasks. Despite their popularity, little is known about how users' preferences and values impact agents' reasoning and behavior. In this work, we investigate how both explicit and implicit user preferences, as well as the underlying user values, influence agent decision-making and action trajectories. We built a controlled testbed of 14 common interactive web tasks, spanning shopping, travel, dining, and housing, each replicated from real websites and integrated with a low-fidelity LLM-based recommender system. We injected 12 human preferences and values as personas into four state-of-the-art agents and systematically analyzed their task behaviors. Our results show that preference and value-infused prompts consistently guided agents toward outcomes that reflected these preferences and values. While the absence of user preference or value guidance led agents to exhibit a strong efficiency bias and employ shortest-path strategies, their presence steered agents' behavior trajectories through the greater use of corresponding filters and interactive web features. Despite their influence, dominant interface cues, such as discounts and advertisements, frequently overrode these effects, shortening the agents' action trajectories and inducing rationalizations that masked rather than reflected value-consistent reasoning. The contributions of this paper are twofold: (1) an open-source testbed for studying the influence of values in agent behaviors, and (2) an empirical investigation of how user preferences and values shape web agent behaviors.

Figures (7)

• Figure 1: We prompted multiple Large Language Model (LLM)-based web agents under three conditions: task instructions framed by a set of values, preferences, or no value framing (baseline). Agents autonomously completed tasks on replica websites. Colored trajectories show captured reasoning-action sequences (e.g., searches, filter use, product selections). The differences in trajectories show how value and preference prompts lead to distinct reasoning strategies and interaction patterns.
• Figure 2: Screenshots of Grumble, a Yelp replica website, curated as one of the interactive test environments
• Figure 3: Examples of promotional UI cues included in the replicated web environments
• Figure 4: Comparison of agent step counts across experimental factors. (a) shows variation by prompting condition, and (b) shows variation by alignment outcome.
• Figure 5: Each curve shows the kernel density estimate (KDE) of bi-gram cosine similarity between agent trajectories. The within-value (orange) curve represents pairs of trajectories with the same values or preferences, while the across-value curve (blue) compares action trajectories by different values. The within-value distribution is shifted slightly to the right, showing higher average action overlap (M=0.17) than across-value (M=0.08, p<0.001).