Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AgentDyn: A Dynamic Open-Ended Benchmark for Evaluating Prompt Injection Attacks of Real-World Agent Security System

Hao Li, Ruoyao Wen, Shanghao Shi, Ning Zhang, Chaowei Xiao

TL;DR

AgentDyn addresses the gap between existing agent-security benchmarks and real-world threat landscapes by introducing a dynamic open-ended benchmark with 60 user tasks and 560 injection cases across Shopping, GitHub, and Daily Life. It emphasizes dynamic planning, the presence of helpful instructions, and longer task trajectories to stress defenses, and evaluates 8 agents against 10 defenses, revealing substantial gaps and over-defensive tendencies. The results show that most defenses perform poorly under AgentDyn’s open-ended conditions, with notable increases in attack success and reductions in utility compared to AgentDojo, demonstrating the need for deployable security strategies. The benchmark provides a practical platform for advancing robust agent-security systems and is available for public use to drive real-world applicability and improvements in defense design.

Abstract

AI agents that autonomously interact with external tools and environments show great promise across real-world applications. However, the external data which agent consumes also leads to the risk of indirect prompt injection attacks, where malicious instructions embedded in third-party content hijack agent behavior. Guided by benchmarks, such as AgentDojo, there has been significant amount of progress in developing defense against the said attacks. As the technology continues to mature, and that agents are increasingly being relied upon for more complex tasks, there is increasing pressing need to also evolve the benchmark to reflect threat landscape faced by emerging agentic systems. In this work, we reveal three fundamental flaws in current benchmarks and push the frontier along these dimensions: (i) lack of dynamic open-ended tasks, (ii) lack of helpful instructions, and (iii) simplistic user tasks. To bridge this gap, we introduce AgentDyn, a manually designed benchmark featuring 60 challenging open-ended tasks and 560 injection test cases across Shopping, GitHub, and Daily Life. Unlike prior static benchmarks, AgentDyn requires dynamic planning and incorporates helpful third-party instructions. Our evaluation of ten state-of-the-art defenses suggests that almost all existing defenses are either not secure enough or suffer from significant over-defense, revealing that existing defenses are still far from real-world deployment. Our benchmark is available at https://github.com/leolee99/AgentDyn.

AgentDyn: A Dynamic Open-Ended Benchmark for Evaluating Prompt Injection Attacks of Real-World Agent Security System

TL;DR

AgentDyn addresses the gap between existing agent-security benchmarks and real-world threat landscapes by introducing a dynamic open-ended benchmark with 60 user tasks and 560 injection cases across Shopping, GitHub, and Daily Life. It emphasizes dynamic planning, the presence of helpful instructions, and longer task trajectories to stress defenses, and evaluates 8 agents against 10 defenses, revealing substantial gaps and over-defensive tendencies. The results show that most defenses perform poorly under AgentDyn’s open-ended conditions, with notable increases in attack success and reductions in utility compared to AgentDojo, demonstrating the need for deployable security strategies. The benchmark provides a practical platform for advancing robust agent-security systems and is available for public use to drive real-world applicability and improvements in defense design.

Abstract

AI agents that autonomously interact with external tools and environments show great promise across real-world applications. However, the external data which agent consumes also leads to the risk of indirect prompt injection attacks, where malicious instructions embedded in third-party content hijack agent behavior. Guided by benchmarks, such as AgentDojo, there has been significant amount of progress in developing defense against the said attacks. As the technology continues to mature, and that agents are increasingly being relied upon for more complex tasks, there is increasing pressing need to also evolve the benchmark to reflect threat landscape faced by emerging agentic systems. In this work, we reveal three fundamental flaws in current benchmarks and push the frontier along these dimensions: (i) lack of dynamic open-ended tasks, (ii) lack of helpful instructions, and (iii) simplistic user tasks. To bridge this gap, we introduce AgentDyn, a manually designed benchmark featuring 60 challenging open-ended tasks and 560 injection test cases across Shopping, GitHub, and Daily Life. Unlike prior static benchmarks, AgentDyn requires dynamic planning and incorporates helpful third-party instructions. Our evaluation of ten state-of-the-art defenses suggests that almost all existing defenses are either not secure enough or suffer from significant over-defense, revealing that existing defenses are still far from real-world deployment. Our benchmark is available at https://github.com/leolee99/AgentDyn.
Paper Structure (19 sections, 4 figures, 7 tables)

This paper contains 19 sections, 4 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work and Preliminaries
  3. Benchmarks of Agent Security
  4. Existing Benchmark Statistics
  5. AgentDyn: An Open-ended Dynamic Agent Security Benchmark
  6. Overview and Structure.
  7. Test Case Generation
  8. AgentDyn Statistics
  9. Experiments
  10. Experiment Setup
  11. Agents and Defenses Evaluation
  12. Comparing with AgentDojo
  13. Analysis of Task Trajectory Length
  14. Conclusion
  15. Limitations
  16. ...and 4 more sections

Figures (4)

  • Figure 1: The Attacked Utility and ASR comparison of 9 advanced defenses powered by GPT-4o on AgentDyn.
  • Figure 2: Comparison between AgentDojo and AgentDyn on four GPT-4o powered defenses, as well as Meta SecAlign.
  • Figure 3: Utility and ASR against the task trajectory length on Vannila GPT-4o.
  • Figure 4: A dynamic open-ended task illustration. Helpful instructions from the environment are highlighted in green.