Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AgentVigil: Generic Black-Box Red-teaming for Indirect Prompt Injection against LLM Agents

Zhun Wang, Vincent Siu, Zhe Ye, Tianneng Shi, Yuzhou Nie, Xuandong Zhao, Chenguang Wang, Wenbo Guo, Dawn Song

TL;DR

<3-5 sentence high-level summary> AgentVigil addresses the vulnerability of black-box LLM agents to indirect prompt injection by introducing a generic fuzzing framework that combines a high-quality seed corpus, adaptive seed scoring, and an MCTS-based seed selector to automatically discover vulnerabilities across diverse agent architectures. It demonstrates strong attack performance on AgentDojo and VWA-adv benchmarks (71% and 70% ASR, respectively), with notable transferability to unseen tasks and models and successful real-world case studies. The framework also reveals weaknesses in existing defenses, showing that conventional mitigations struggle against robust, automatically generated adversarial prompts. Together, these findings highlight urgent security gaps in agent systems and provide a practical, extensible baseline for evaluating and strengthening defenses in real-world deployments.

Abstract

The strong planning and reasoning capabilities of Large Language Models (LLMs) have fostered the development of agent-based systems capable of leveraging external tools and interacting with increasingly complex environments. However, these powerful features also introduce a critical security risk: indirect prompt injection, a sophisticated attack vector that compromises the core of these agents, the LLM, by manipulating contextual information rather than direct user prompts. In this work, we propose a generic black-box fuzzing framework, AgentVigil, designed to automatically discover and exploit indirect prompt injection vulnerabilities across diverse LLM agents. Our approach starts by constructing a high-quality initial seed corpus, then employs a seed selection algorithm based on Monte Carlo Tree Search (MCTS) to iteratively refine inputs, thereby maximizing the likelihood of uncovering agent weaknesses. We evaluate AgentVigil on two public benchmarks, AgentDojo and VWA-adv, where it achieves 71% and 70% success rates against agents based on o3-mini and GPT-4o, respectively, nearly doubling the performance of baseline attacks. Moreover, AgentVigil exhibits strong transferability across unseen tasks and internal LLMs, as well as promising results against defenses. Beyond benchmark evaluations, we apply our attacks in real-world environments, successfully misleading agents to navigate to arbitrary URLs, including malicious sites.

AgentVigil: Generic Black-Box Red-teaming for Indirect Prompt Injection against LLM Agents

TL;DR

<3-5 sentence high-level summary> AgentVigil addresses the vulnerability of black-box LLM agents to indirect prompt injection by introducing a generic fuzzing framework that combines a high-quality seed corpus, adaptive seed scoring, and an MCTS-based seed selector to automatically discover vulnerabilities across diverse agent architectures. It demonstrates strong attack performance on AgentDojo and VWA-adv benchmarks (71% and 70% ASR, respectively), with notable transferability to unseen tasks and models and successful real-world case studies. The framework also reveals weaknesses in existing defenses, showing that conventional mitigations struggle against robust, automatically generated adversarial prompts. Together, these findings highlight urgent security gaps in agent systems and provide a practical, extensible baseline for evaluating and strengthening defenses in real-world deployments.

Abstract

The strong planning and reasoning capabilities of Large Language Models (LLMs) have fostered the development of agent-based systems capable of leveraging external tools and interacting with increasingly complex environments. However, these powerful features also introduce a critical security risk: indirect prompt injection, a sophisticated attack vector that compromises the core of these agents, the LLM, by manipulating contextual information rather than direct user prompts. In this work, we propose a generic black-box fuzzing framework, AgentVigil, designed to automatically discover and exploit indirect prompt injection vulnerabilities across diverse LLM agents. Our approach starts by constructing a high-quality initial seed corpus, then employs a seed selection algorithm based on Monte Carlo Tree Search (MCTS) to iteratively refine inputs, thereby maximizing the likelihood of uncovering agent weaknesses. We evaluate AgentVigil on two public benchmarks, AgentDojo and VWA-adv, where it achieves 71% and 70% success rates against agents based on o3-mini and GPT-4o, respectively, nearly doubling the performance of baseline attacks. Moreover, AgentVigil exhibits strong transferability across unseen tasks and internal LLMs, as well as promising results against defenses. Beyond benchmark evaluations, we apply our attacks in real-world environments, successfully misleading agents to navigate to arbitrary URLs, including malicious sites.
Paper Structure (38 sections, 4 figures, 5 tables, 3 algorithms)

This paper contains 38 sections, 4 figures, 5 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Our approach.
  3. Results.
  4. Related Work
  5. LLM agents.
  6. Existing attacks.
  7. Existing defenses.
  8. Threat Model
  9. Blackbox setting of the agent systems.
  10. User assumptions.
  11. Attacker's capabilities and goals.
  12. Method
  13. Overview
  14. Corpus Collection
  15. Mutation Design
  16. ...and 23 more sections

Figures (4)

  • Figure 1: An example of deceiving a web agent through indirect prompt injection in a customer review on the shopping website. The user requests the agent to find a screen protector and list out reviewers who mention about good fingerprint resistant, but the adversarial prompt redirects the agent to arbitrary URLs specified in the injected text, potentially leading to unrelated sites, phishing sites, malware downloads, or exposure of private data. We achieve the attack with other URLs such as phishing sites, malware downloads, queries with privacy leakage to verify the severity.
  • Figure 2: Architecture of AgentVigil and typical process of an indirect prompt injection attack. AgentVigil systematically enhances indirect prompt injection attacks by iteratively refining adversarial prompts. It begins with a high-quality initial corpus of prompt templates, which are tested across various injection tasks to generate initial seeds. Through an iterative fuzzing loop, a Monte Carlo Tree Search (MCTS)-based seed selector identifies promising seeds, a mutator applies transformations, and the modified prompts are evaluated based on attack success and task coverage by the scorer. This adaptive approach ensures scalability and effectiveness across diverse agent architectures and tasks.
  • Figure 3: Coverage over fuzzing iteration steps achieved by AgentVigil (the solid line) on AgentDojo with two ablation settings (the dashed lines): (1) without the high-quality initial corpus, (2) without the adaptive seed scoring strategy and the MCTS-based seed selection.
  • Figure 4: Coverage over fuzzing iterations achieved by AgentVigil on VWA-adv.