Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Securing AI Agents Against Prompt Injection Attacks

Badrinath Ramakrishnan, Akshaya Balaji

TL;DR

Prompt injection in retrieval-augmented generation enables adversaries to manipulate model behavior via retrieved content. The paper introduces a comprehensive benchmark of 847 test cases and a multi-layer defense combining content filtering, hierarchical prompt guardrails, and response verification, evaluated across seven LLMs. Results show attack success dropping from 73.2% to 8.7% while preserving 94.3% of legitimate functionality, establishing practical safety improvements and a standardized evaluation framework. These contributions support safer deployment of AI agents in adversarial environments and provide reusable datasets and implementations for future research.

Abstract

Retrieval-augmented generation (RAG) systems have become widely used for enhancing large language model capabilities, but they introduce significant security vulnerabilities through prompt injection attacks. We present a comprehensive benchmark for evaluating prompt injection risks in RAG-enabled AI agents and propose a multi-layered defense framework. Our benchmark includes 847 adversarial test cases across five attack categories: direct injection, context manipulation, instruction override, data exfiltration, and cross-context contamination. We evaluate three defense mechanisms: content filtering with embedding-based anomaly detection, hierarchical system prompt guardrails, and multi-stage response verification, across seven state-of-the-art language models. Our combined framework reduces successful attack rates from 73.2% to 8.7% while maintaining 94.3% of baseline task performance. We release our benchmark dataset and defense implementation to support future research in AI agent security.

Securing AI Agents Against Prompt Injection Attacks

TL;DR

Prompt injection in retrieval-augmented generation enables adversaries to manipulate model behavior via retrieved content. The paper introduces a comprehensive benchmark of 847 test cases and a multi-layer defense combining content filtering, hierarchical prompt guardrails, and response verification, evaluated across seven LLMs. Results show attack success dropping from 73.2% to 8.7% while preserving 94.3% of legitimate functionality, establishing practical safety improvements and a standardized evaluation framework. These contributions support safer deployment of AI agents in adversarial environments and provide reusable datasets and implementations for future research.

Abstract

Retrieval-augmented generation (RAG) systems have become widely used for enhancing large language model capabilities, but they introduce significant security vulnerabilities through prompt injection attacks. We present a comprehensive benchmark for evaluating prompt injection risks in RAG-enabled AI agents and propose a multi-layered defense framework. Our benchmark includes 847 adversarial test cases across five attack categories: direct injection, context manipulation, instruction override, data exfiltration, and cross-context contamination. We evaluate three defense mechanisms: content filtering with embedding-based anomaly detection, hierarchical system prompt guardrails, and multi-stage response verification, across seven state-of-the-art language models. Our combined framework reduces successful attack rates from 73.2% to 8.7% while maintaining 94.3% of baseline task performance. We release our benchmark dataset and defense implementation to support future research in AI agent security.

Paper Structure

This paper contains 27 sections, 1 equation, 2 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background and Threat Model
  3. Retrieval-Augmented Generation Architecture
  4. Threat Model
  5. Attack Categories
  6. Related Work
  7. Benchmark Design
  8. Benchmark Design
  9. Dataset Construction
  10. Sophistication Levels
  11. Evaluation Metrics
  12. Defense Framework
  13. Content Filtering with Embedding Analysis
  14. Hierarchical System Prompt Guardrails
  15. Multi-Stage Response Verification
  16. ...and 12 more sections

Figures (2)

  • Figure 1: Multi-layered defense framework architecture. Retrieved content passes through embedding analysis, content filtering, guardrail application, and response verification before output.
  • Figure 2: Attack success rates across models with and without defense framework. All models benefit substantially from defenses, though baseline vulnerability varies significantly.