Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Indirect Prompt Injections: Are Firewalls All You Need, or Stronger Benchmarks?

Rishika Bhagwatkar, Kevin Kasa, Abhay Puri, Gabriel Huang, Irina Rish, Graham W. Taylor, Krishnamurthy Dj Dvijotham, Alexandre Lacoste

TL;DR

Indirect prompt injection (IPI) poses a critical risk for tool-calling LLM agents by hiding malicious instructions in external content. The authors introduce a minimal, modular firewall framework consisting of a Tool-Input Minimizer and a Tool-Output Sanitizer that operate at the agent–tool boundary and require no LLM retraining. Across four benchmarks (AgentDojo, Agent Security Bench, InjecAgent, Tau-Bench), the approach yields $0\%$ attack rate with high task utility and offers superior security-utility tradeoffs versus prior defenses. The paper also critiques current benchmarks, proposes fixes for more faithful evaluation, and argues for stronger, diverse adaptive attacks to drive robust progress.

Abstract

AI agents are vulnerable to indirect prompt injection attacks, where malicious instructions embedded in external content or tool outputs cause unintended or harmful behavior. Inspired by the well-established concept of firewalls, we show that a simple, modular and model-agnostic defense operating at the agent--tool interface achieves perfect security (0% or the lowest possible attack success rate) with high utility (task success rate) across four public benchmarks: AgentDojo, Agent Security Bench, InjecAgent and tau-Bench, while achieving a state-of-the-art security-utility tradeoff compared to prior results. Specifically, we employ a defense based on two firewalls: a Tool-Input Firewall (Minimizer) and a Tool-Output Firewall (Sanitizer). Unlike prior complex approaches, this firewall defense makes minimal assumptions on the agent and can be deployed out-of-the-box, while maintaining strong performance without compromising utility. However, our analysis also reveals critical limitations in these existing benchmarks, including flawed success metrics, implementation bugs, and most importantly, weak attacks, hindering significant progress in the field. To foster more meaningful progress, we present targeted fixes to these issues for AgentDojo and Agent Security Bench while proposing best-practices for more robust benchmark design. Further, we demonstrate that although these firewalls push the state-of-the-art on existing benchmarks, it is still possible to bypass them in practice, underscoring the need to incorporate stronger attacks in security benchmarks. Overall, our work shows that existing agentic security benchmarks are easily saturated by a simple approach and highlights the need for stronger agentic security benchmarks with carefully chosen evaluation metrics and strong adaptive attacks.

Indirect Prompt Injections: Are Firewalls All You Need, or Stronger Benchmarks?

TL;DR

Indirect prompt injection (IPI) poses a critical risk for tool-calling LLM agents by hiding malicious instructions in external content. The authors introduce a minimal, modular firewall framework consisting of a Tool-Input Minimizer and a Tool-Output Sanitizer that operate at the agent–tool boundary and require no LLM retraining. Across four benchmarks (AgentDojo, Agent Security Bench, InjecAgent, Tau-Bench), the approach yields attack rate with high task utility and offers superior security-utility tradeoffs versus prior defenses. The paper also critiques current benchmarks, proposes fixes for more faithful evaluation, and argues for stronger, diverse adaptive attacks to drive robust progress.

Abstract

AI agents are vulnerable to indirect prompt injection attacks, where malicious instructions embedded in external content or tool outputs cause unintended or harmful behavior. Inspired by the well-established concept of firewalls, we show that a simple, modular and model-agnostic defense operating at the agent--tool interface achieves perfect security (0% or the lowest possible attack success rate) with high utility (task success rate) across four public benchmarks: AgentDojo, Agent Security Bench, InjecAgent and tau-Bench, while achieving a state-of-the-art security-utility tradeoff compared to prior results. Specifically, we employ a defense based on two firewalls: a Tool-Input Firewall (Minimizer) and a Tool-Output Firewall (Sanitizer). Unlike prior complex approaches, this firewall defense makes minimal assumptions on the agent and can be deployed out-of-the-box, while maintaining strong performance without compromising utility. However, our analysis also reveals critical limitations in these existing benchmarks, including flawed success metrics, implementation bugs, and most importantly, weak attacks, hindering significant progress in the field. To foster more meaningful progress, we present targeted fixes to these issues for AgentDojo and Agent Security Bench while proposing best-practices for more robust benchmark design. Further, we demonstrate that although these firewalls push the state-of-the-art on existing benchmarks, it is still possible to bypass them in practice, underscoring the need to incorporate stronger attacks in security benchmarks. Overall, our work shows that existing agentic security benchmarks are easily saturated by a simple approach and highlights the need for stronger agentic security benchmarks with carefully chosen evaluation metrics and strong adaptive attacks.

Paper Structure

This paper contains 47 sections, 1 figure, 20 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Method
  4. Task Setup
  5. Threat Model
  6. Defense Objective
  7. Firewall Defenses
  8. Experiments
  9. Setup
  10. (1) AgentDojo:
  11. (2) Agent Security Bench:
  12. (3) InjecAgent:
  13. (4) Tau-bench:
  14. Evaluation Metrics
  15. Results
  16. ...and 32 more sections

Figures (1)

  • Figure 1: Minimize & Sanitize tool-calling pipeline. The user (trusted) gives a task to the AI agent. The agent generates tool calls, which are passed through the Minimizer to strip any information that is not required or relevant for the given user task, executed, and finally passed through the Sanitizer to remove all suspected prompt injections content, before returning the result to the agent.