Defending Against Prompt Injection with DataFilter

TL;DR

The paper tackles prompt injection as a critical security risk for LLM-enabled agents by introducing DataFilter, a test-time, model-agnostic data sanitizer trained via supervised fine-tuning to remove malicious injections while preserving benign content. DataFilter ingests a trusted prompt and untrusted data, producing a cleaned input that prevents attacker instructions from influencing the backend LLM, and it is designed to be easily deployed with any model. Across benchmarks including SEP, InjecAgent, AgentDojo, and AlpacaEval2, DataFilter dramatically reduces attack success rates to near zero with only about a 1% utility loss, outperforming detector- and prompting-based defenses and avoiding the heavier trade-offs of system- or model-level approaches. The solution is plug-and-play for black-box LLMs, offering a practical, broadly applicable defense that enhances security without sacrificing performance, with the caveat of potential overhead and future resilience against optimization-based adaptive attacks.

Abstract

When large language model (LLM) agents are increasingly deployed to automate tasks and interact with untrusted external data, prompt injection emerges as a significant security threat. By injecting malicious instructions into the data that LLMs access, an attacker can arbitrarily override the original user task and redirect the agent toward unintended, potentially harmful actions. Existing defenses either require access to model weights (fine-tuning), incur substantial utility loss (detection-based), or demand non-trivial system redesign (system-level). Motivated by this, we propose DataFilter, a test-time model-agnostic defense that removes malicious instructions from the data before it reaches the backend LLM. DataFilter is trained with supervised fine-tuning on simulated injections and leverages both the user's instruction and the data to selectively strip adversarial content while preserving benign information. Across multiple benchmarks, DataFilter consistently reduces the prompt injection attack success rates to near zero while maintaining the LLMs' utility. DataFilter delivers strong security, high utility, and plug-and-play deployment, making it a strong practical defense to secure black-box commercial LLMs against prompt injection. Our DataFilter model is released at https://huggingface.co/JoyYizhu/DataFilter for immediate use, with the code to reproduce our results at https://github.com/yizhu-joy/DataFilter.

Figures (4)

• Figure 1: DataFilter takes both the trusted instruction and untrusted data as input, removes potential prompt injections, and outputs the sanitized data. The backend LLM then executes the original instruction using the sanitized data.
• Figure 2: DataFilter achieves a better tradeoff between security (Attack Success Rate, ASR, $\downarrow$) and utility ($\uparrow$) than any prior defense. The star indicates the best one could hope for (zero ASR without utility drop). DataFilter approaches this ideal more closely than other defenses. The ASR scores are averaged across three benchmarks: SEP Zeverev2023can, InjecAgent zhan2024injecagent, and AgentDojo debenedetti2024agentdojo. The ASR for a benchmark is calculated by the maximum ASR of various attacks (SEP, InjecAgent, and AgentDojo are tested with 6, 2, and 4 attack methods, respectively). The utility scores are averages across two benchmarks: AlpacaEval2 alpaca_eval and AgentDojo debenedetti2024agentdojo. SEP and AlpacaEval2 are for instruction following; InjecAgent and AgentDojo are for agentic tool-calling.
• Figure 3: Utility–security trade-offs on AgentDojo. The star indicates the best defense could hope for (zero ASR without utility drop). DataFilter approaches this ideal more closely than all other tested defenses. The utility is tested without any attack. The ASR is the maximum ASR of 4 tested attacks on AgentDojo.
• Figure 4: Visualization of numbers in \ref{['tab:SEP-merged']}: ASR ($\downarrow$) on SEP against 6 attacks.