Investigating the Impact of Dark Patterns on LLM-Based Web Agents

TL;DR

This work delivers the first systematic evaluation of dark patterns on LLM-based web agents by introducing LiteAgent, an agent-agnostic logging framework, and TrickyArena, a controlled web environment with configurable dark patterns. Across six agents and three LLMs, the study demonstrates substantial susceptibility to dark patterns, with single patterns causing about a $41\%$ average likelihood of manipulation and increased vulnerability when patterns stack or when UI attributes change. It also shows that factors such as LLM choice, agent architecture, observation modality, and targeted prompting influence both task success and deception risk, and investigates initial countermeasures like specific postscripts with partial success. The findings underscore the need for defense-in-depth, combining web-safety measures, DP-aware planning, and potential pattern-removal tools to robustly guard LLM-driven web automation in real-world scenarios.

Abstract

As users increasingly turn to large language model (LLM) based web agents to automate online tasks, agents may encounter dark patterns: deceptive user interface designs that manipulate users into making unintended decisions. Although dark patterns primarily target human users, their potentially harmful impacts on LLM-based generalist web agents remain unexplored. In this paper, we present the first study that investigates the impact of dark patterns on the decision-making process of LLM-based generalist web agents. To achieve this, we introduce LiteAgent, a lightweight framework that automatically prompts agents to execute tasks while capturing comprehensive logs and screen-recordings of their interactions. We also present TrickyArena, a controlled environment comprising web applications from domains such as e-commerce, streaming services, and news platforms, each containing diverse and realistic dark patterns that can be selectively enabled or disabled. Using LiteAgent and TrickyArena, we conduct multiple experiments to assess the impact of both individual and combined dark patterns on web agent behavior. We evaluate six popular LLM-based generalist web agents across three LLMs and discover that when there is a single dark pattern present, agents are susceptible to it an average of 41% of the time. We also find that modifying dark pattern UI attributes through visual design changes or HTML code adjustments and introducing multiple dark patterns simultaneously can influence agent susceptibility. This study emphasizes the need for holistic defense mechanisms in web agents, encompassing both agent-specific protections and broader web safety measures.

Figures (15)

• Figure 1: Methodology overview for evaluating web agent susceptibility to dark patterns. A web agent and a scenario from TrickyArena are selected. The agent is run on the scenario, and its actions are recorded. These actions are compared against pre-defined criteria to determine task completion and whether the agent fell for the DPs.
• Figure 2: Example scenario where the task is to buy the best-reviewed water bottle on an e-commerce site with two dark patterns enabled. The first forcibly adds a warranty to the cart without explicit consent, and the second is a cookie preference pop-up.
• Figure 3: LiteAgent functionality overview. A scenario and agent is given to LiteAgent. LiteAgent initializes the scenario environment by loading the agent and launching the browser instance. Event listeners are injected into the browser and the agent is given the scenario task. As the agent interacts with the browser, the event listeners log all the actions. When the agent is finished, an action trace and session video are returned.
• Figure 4: LiteAgent's event listener injection and action logging pipeline. LiteAgent injects JavaScript event listeners into the browser's context and exposes Python logging functions to these listeners. When an event listener is triggered by an action, the Python logging function is called and logs the action into an action database.
• Figure 5: Overall Task Success Rate (TSR) comparison of no dark pattern vs. one dark pattern enabled.