Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PromptShield: Deployable Detection for Prompt Injection Attacks

Dennis Jacob, Hend Alzahrani, Zhanhao Hu, Basel Alomair, David Wagner

TL;DR

The paper tackles the practical challenge of detecting prompt injection in LLM-based applications by distinguishing conversational from application-structured data and proposing deployment-focused evaluation. It introduces PromptShield, a curated benchmark and a high-performing detector trained on its train split, with a deployment scheme that targets ultra-low FPRs and demonstrates strong low-FPR detection across architectures. Key findings show that data curation and larger, well-chosen models substantially boost detection, achieving up to $TPR=94.8\%$ at $FPR=1\%$ and $TPR=65.3\%$ at $FPR=0.1\%$ for different setups, significantly beating prior detectors like PromptGuard. The authors release the benchmark and code publicly to spur practical, scalable defenses and encourage further research into data-driven improvements and deployment-aware evaluation for prompt-injection defenses.

Abstract

Application designers have moved to integrate large language models (LLMs) into their products. However, many LLM-integrated applications are vulnerable to prompt injections. While attempts have been made to address this problem by building prompt injection detectors, many are not yet suitable for practical deployment. To support research in this area, we introduce PromptShield, a benchmark for training and evaluating deployable prompt injection detectors. Our benchmark is carefully curated and includes both conversational and application-structured data. In addition, we use insights from our curation process to fine-tune a new prompt injection detector that achieves significantly higher performance in the low false positive rate (FPR) evaluation regime compared to prior schemes. Our work suggests that careful curation of training data and larger models can contribute to strong detector performance.

PromptShield: Deployable Detection for Prompt Injection Attacks

TL;DR

The paper tackles the practical challenge of detecting prompt injection in LLM-based applications by distinguishing conversational from application-structured data and proposing deployment-focused evaluation. It introduces PromptShield, a curated benchmark and a high-performing detector trained on its train split, with a deployment scheme that targets ultra-low FPRs and demonstrates strong low-FPR detection across architectures. Key findings show that data curation and larger, well-chosen models substantially boost detection, achieving up to at and at for different setups, significantly beating prior detectors like PromptGuard. The authors release the benchmark and code publicly to spur practical, scalable defenses and encourage further research into data-driven improvements and deployment-aware evaluation for prompt-injection defenses.

Abstract

Application designers have moved to integrate large language models (LLMs) into their products. However, many LLM-integrated applications are vulnerable to prompt injections. While attempts have been made to address this problem by building prompt injection detectors, many are not yet suitable for practical deployment. To support research in this area, we introduce PromptShield, a benchmark for training and evaluating deployable prompt injection detectors. Our benchmark is carefully curated and includes both conversational and application-structured data. In addition, we use insights from our curation process to fine-tune a new prompt injection detector that achieves significantly higher performance in the low false positive rate (FPR) evaluation regime compared to prior schemes. Our work suggests that careful curation of training data and larger models can contribute to strong detector performance.
Paper Structure (48 sections, 3 figures, 12 tables)

This paper contains 48 sections, 3 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. LLM-integrated applications
  4. Prompt injection attacks
  5. Prompt injection detectors
  6. A taxonomy for LLM requests
  7. Design Framework
  8. PromptShield benchmark
  9. Benign data
  10. Conversational data
  11. Application-structured data
  12. Injection data
  13. Injection generation methods
  14. Naturally occurring injections
  15. Training/evaluation split.
  16. ...and 33 more sections

Figures (3)

  • Figure 1: PromptShield for prompt injection detection. Realistic deployment settings require the ability to handle both conversational data and application-structured data. We propose a novel benchmark that more accurately captures this reality and train a detection model that achieves strong detection performance.
  • Figure 2: Our scheme performs far better than all prior detectors on the evaluation split of our benchmark. Each bar shows the TPR achieved, at 0.1% FPR; our scheme achieves 65% TPR, compared to 9% for the best prior model.
  • Figure 3: Deployment scheme for the PromptShield detector. In the left panel we obtain raw output scores. In the middle panel we construct the ROC curve (in grey box) by sweeping across a range of threshold values. Finally, we use interpolation to find thresholds that result in FPRs close to our targets; we deploy the model with the chosen threshold in the right panel.