Can Indirect Prompt Injection Attacks Be Detected and Removed?

TL;DR

The paper addresses indirect prompt injection attacks by evaluating both detection and removal of malicious instructions embedded in external documents read by LLMs. It builds Inj-SQuAD and Inj-TriviaQA benchmarks with crafted injections and trains detectors on dedicated data, also proposing segmentation and extraction removal methods to mitigate injected content. Key findings show existing instructed LLMs and many open-source detectors struggle with indirect attacks; specially trained detectors perform well but can exhibit over-defense and position-generalization issues. Removal methods can reduce injected content, with segmentation generally outperforming extraction except for tail injections, and a combined detection+removal filtering approach offering effective defense though not universally robust across all attack types. The work provides practical insights and a foundation for improving indirect-prompt-injection defenses, highlighting the need for robust, position-agnostic removal techniques and broader evaluation across attack variants.

Abstract

Prompt injection attacks manipulate large language models (LLMs) by misleading them to deviate from the original input instructions and execute maliciously injected instructions, because of their instruction-following capabilities and inability to distinguish between the original input instructions and maliciously injected instructions. To defend against such attacks, recent studies have developed various detection mechanisms. If we restrict ourselves specifically to works which perform detection rather than direct defense, most of them focus on direct prompt injection attacks, while there are few works for the indirect scenario, where injected instructions are indirectly from external tools, such as a search engine. Moreover, current works mainly investigate injection detection methods and pay less attention to the post-processing method that aims to mitigate the injection after detection. In this paper, we investigate the feasibility of detecting and removing indirect prompt injection attacks, and we construct a benchmark dataset for evaluation. For detection, we assess the performance of existing LLMs and open-source detection models, and we further train detection models using our crafted training datasets. For removal, we evaluate two intuitive methods: (1) the segmentation removal method, which segments the injected document and removes parts containing injected instructions, and (2) the extraction removal method, which trains an extraction model to identify and remove injected instructions.

Figures (6)

• Figure 1: (a) represents a direct prompt injection attack example and (b) illustrates an indirect prompt injection attack example.
• Figure 2: The pipeline for the filtering method.
• Figure 3: The defense performance and over-defense problem are a trade-off for models trained on crafted SQuAD training dataset and evaluated on Inj-TriviaQA documents. We report the minimum performance across different attacks and positions. The evaluation metric is the false positive rate for over-defense and true positive rate for normal defense.
• Figure 4: The segmentation removal process.
• Figure 5: The extraction removal process.