Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents

Hanrong Zhang, Jingyuan Huang, Kai Mei, Yifei Yao, Zhenting Wang, Chenlu Zhan, Hongwei Wang, Yongfeng Zhang

TL;DR

ASB addresses the security evaluation gap for LLM-based agents by formalizing a broad taxonomy of attacks (DPI, IPI, Memory Poisoning, PoT Backdoor, and Mixed) and defenses across 10 scenarios and 13 backbones, using 7 metrics. It demonstrates significant vulnerabilities, with an average $ASR$ up to $84.30\%$ and limited defense effectiveness, while introducing the Net Resilient Performance (NRP) metric to balance utility and security. Key contributions include the first holistic benchmark for LLM agents, a novel PoT backdoor, and comprehensive taxonomies and evaluation pipelines that guide defense research and backbone selection. The work has practical impact by providing reproducible benchmarks and code, emphasizing the need for stronger, multi-faceted defenses in real-world agent deployments.

Abstract

Although LLM-based agents, powered by Large Language Models (LLMs), can use external tools and memory mechanisms to solve complex real-world tasks, they may also introduce critical security vulnerabilities. However, the existing literature does not comprehensively evaluate attacks and defenses against LLM-based agents. To address this, we introduce Agent Security Bench (ASB), a comprehensive framework designed to formalize, benchmark, and evaluate the attacks and defenses of LLM-based agents, including 10 scenarios (e.g., e-commerce, autonomous driving, finance), 10 agents targeting the scenarios, over 400 tools, 27 different types of attack/defense methods, and 7 evaluation metrics. Based on ASB, we benchmark 10 prompt injection attacks, a memory poisoning attack, a novel Plan-of-Thought backdoor attack, 4 mixed attacks, and 11 corresponding defenses across 13 LLM backbones. Our benchmark results reveal critical vulnerabilities in different stages of agent operation, including system prompt, user prompt handling, tool usage, and memory retrieval, with the highest average attack success rate of 84.30\%, but limited effectiveness shown in current defenses, unveiling important works to be done in terms of agent security for the community. We also introduce a new metric to evaluate the agents' capability to balance utility and security. Our code can be found at https://github.com/agiresearch/ASB.

Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents

TL;DR

ASB addresses the security evaluation gap for LLM-based agents by formalizing a broad taxonomy of attacks (DPI, IPI, Memory Poisoning, PoT Backdoor, and Mixed) and defenses across 10 scenarios and 13 backbones, using 7 metrics. It demonstrates significant vulnerabilities, with an average up to and limited defense effectiveness, while introducing the Net Resilient Performance (NRP) metric to balance utility and security. Key contributions include the first holistic benchmark for LLM agents, a novel PoT backdoor, and comprehensive taxonomies and evaluation pipelines that guide defense research and backbone selection. The work has practical impact by providing reproducible benchmarks and code, emphasizing the need for stronger, multi-faceted defenses in real-world agent deployments.

Abstract

Although LLM-based agents, powered by Large Language Models (LLMs), can use external tools and memory mechanisms to solve complex real-world tasks, they may also introduce critical security vulnerabilities. However, the existing literature does not comprehensively evaluate attacks and defenses against LLM-based agents. To address this, we introduce Agent Security Bench (ASB), a comprehensive framework designed to formalize, benchmark, and evaluate the attacks and defenses of LLM-based agents, including 10 scenarios (e.g., e-commerce, autonomous driving, finance), 10 agents targeting the scenarios, over 400 tools, 27 different types of attack/defense methods, and 7 evaluation metrics. Based on ASB, we benchmark 10 prompt injection attacks, a memory poisoning attack, a novel Plan-of-Thought backdoor attack, 4 mixed attacks, and 11 corresponding defenses across 13 LLM backbones. Our benchmark results reveal critical vulnerabilities in different stages of agent operation, including system prompt, user prompt handling, tool usage, and memory retrieval, with the highest average attack success rate of 84.30\%, but limited effectiveness shown in current defenses, unveiling important works to be done in terms of agent security for the community. We also introduce a new metric to evaluate the agents' capability to balance utility and security. Our code can be found at https://github.com/agiresearch/ASB.
Paper Structure (63 sections, 21 equations, 4 figures, 21 tables)

This paper contains 63 sections, 21 equations, 4 figures, 21 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Definitions to Basic Concepts and Threat Model
  4. Defining Basic Concepts
  5. Threat Model
  6. Formalizing Attacks and Defenses in LLM Agents
  7. Formalizing Prompt Injection Attacks
  8. Direct Prompt Injection Attacks
  9. Indirect Prompt Injection Attacks
  10. Attack Framework for Different Prompt Injection Ways
  11. Formalizing Memory Poisoning Attack
  12. Detailed Adversarial Goal
  13. Attack Framework
  14. Formalizing Plan-of-Thought Backdoor Attack
  15. Detailed Adversarial Goal
  16. ...and 48 more sections

Figures (4)

  • Figure 1: Overview of the LLM Agent Attacking Framework, including Direct Prompt Injections (DPI), Indirect Prompt Injections (IPI), Plan-of-Thought (PoT) Backdoor, and Memory Poisoning Attacks, which target the user query, observations, system prompts, and memory retrieval respectively of the agent during action planning and execution.
  • Figure 2: Visual comparisons between PNA vs ASR, LLM Capability vs PNA and LLM Capability vs ASR.
  • Figure 3: Illustration of four attack types targeting LLM agents. Direct Prompt Injections (DPI) manipulate the user prompt, Indirect Prompt Injections (IPI) alter observation data to interfere with later actions, Plan-of-Thought (PoT) Backdoor Attack triggers hidden actions upon specific inputs, and Memory Poisoning Attack injects malicious plans into the agent’s memory, causing the agent to utilize attacker-specified tools.
  • Figure 4: FPR vs. FNR curve for PPL detection in identifying memory poisoning attack. High perplexity indicates compromised content. The curve shows FNR and FPR variations across different thresholds. Shallower colors correspond to lower thresholds, while darker colors correspond to higher thresholds.