Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Too Helpful to Be Safe: User-Mediated Attacks on Planning and Web-Use Agents

Fengchao Chen, Tingmin Wu, Van Nguyen, Carsten Rudolph

TL;DR

This paper studies user-mediated attacks, where benign users are tricked into relaying untrusted or attacker-controlled content to agents, and analyzed how commercial LLM agents respond under such conditions, revealing that agents are too helpful to be safe by default.

Abstract

Large Language Models (LLMs) have enabled agents to move beyond conversation toward end-to-end task execution and become more helpful. However, this helpfulness introduces new security risks stem less from direct interface abuse than from acting on user-provided content. Existing studies on agent security largely focus on model-internal vulnerabilities or adversarial access to agent interfaces, overlooking attacks that exploit users as unintended conduits. In this paper, we study user-mediated attacks, where benign users are tricked into relaying untrusted or attacker-controlled content to agents, and analyze how commercial LLM agents respond under such conditions. We conduct a systematic evaluation of 12 commercial agents in a sandboxed environment, covering 6 trip-planning agents and 6 web-use agents, and compare agent behavior across scenarios with no, soft, and hard user-requested safety checks. Our results show that agents are too helpful to be safe by default. Without explicit safety requests, trip-planning agents bypass safety constraints in over 92% of cases, converting unverified content into confident booking guidance. Web-use agents exhibit near-deterministic execution of risky actions, with 9 out of 17 supported tests reaching a 100% bypass rate. Even when users express soft or hard safety intent, constraint bypass remains substantial, reaching up to 54.7% and 7% for trip-planning agents, respectively. These findings reveal that the primary issue is not a lack of safety capability, but its prioritization. Agents invoke safety checks only conditionally when explicitly prompted, and otherwise default to goal-driven execution. Moreover, agents lack clear task boundaries and stopping rules, frequently over-executing workflows in ways that lead to unnecessary data disclosure and real-world harm.

Too Helpful to Be Safe: User-Mediated Attacks on Planning and Web-Use Agents

TL;DR

This paper studies user-mediated attacks, where benign users are tricked into relaying untrusted or attacker-controlled content to agents, and analyzed how commercial LLM agents respond under such conditions, revealing that agents are too helpful to be safe by default.

Abstract

Large Language Models (LLMs) have enabled agents to move beyond conversation toward end-to-end task execution and become more helpful. However, this helpfulness introduces new security risks stem less from direct interface abuse than from acting on user-provided content. Existing studies on agent security largely focus on model-internal vulnerabilities or adversarial access to agent interfaces, overlooking attacks that exploit users as unintended conduits. In this paper, we study user-mediated attacks, where benign users are tricked into relaying untrusted or attacker-controlled content to agents, and analyze how commercial LLM agents respond under such conditions. We conduct a systematic evaluation of 12 commercial agents in a sandboxed environment, covering 6 trip-planning agents and 6 web-use agents, and compare agent behavior across scenarios with no, soft, and hard user-requested safety checks. Our results show that agents are too helpful to be safe by default. Without explicit safety requests, trip-planning agents bypass safety constraints in over 92% of cases, converting unverified content into confident booking guidance. Web-use agents exhibit near-deterministic execution of risky actions, with 9 out of 17 supported tests reaching a 100% bypass rate. Even when users express soft or hard safety intent, constraint bypass remains substantial, reaching up to 54.7% and 7% for trip-planning agents, respectively. These findings reveal that the primary issue is not a lack of safety capability, but its prioritization. Agents invoke safety checks only conditionally when explicitly prompted, and otherwise default to goal-driven execution. Moreover, agents lack clear task boundaries and stopping rules, frequently over-executing workflows in ways that lead to unnecessary data disclosure and real-world harm.
Paper Structure (22 sections, 7 equations, 5 figures, 9 tables)

This paper contains 22 sections, 7 equations, 5 figures, 9 tables.

Table of Contents

  1. Introduction
  2. The Attack Space for LLM-based Agents
  3. General abstraction for LLM-based agents and users.
  4. Interaction Model.
  5. Attack Strategies.
  6. Our work
  7. Threat Model
  8. Experiment Setting
  9. Test Environment
  10. Attack Design
  11. Evaluation Framework
  12. Breaking Trip-Planning Agents
  13. Case 1: User-Mediated Attack Pipeline
  14. Results
  15. Breaking Commercial Web-Use Agents
  16. ...and 7 more sections

Figures (5)

  • Figure 1: Threat Model of attacks when users are using commercial agents.
  • Figure 2: Unsafe User-side boundaries in trip-planning agents. Benign users forward attacker-seeded content into the agent, which then legitimizes unverified resources and converts them into actionable guidance.
  • Figure 3: Representative failure cases of trip-planning agents under user-mediated attack.
  • Figure 4: Unsafe User-side boundaries in Web-use Agents (WebUAs). Benign users forward attacker-seeded tasks, which WebUAs execute by following interface cues while overlooking security-relevant context and execution necessity.
  • Figure 5: Representative execution failures of WebUAs. Red boxes highlight risky or policy-violating behaviors; blue and green boxes denote standard execution flow.