AlignSentinel: Alignment-Aware Detection of Prompt Injection Attacks

TL;DR

Prompt injection threatens LLMs by embedding malicious instructions into inputs. AlignSentinel reframes detection as a three-class problem—misaligned, aligned, and non-instruction inputs—using attention-based signals to distinguish how inputs interact with higher-priority instructions. It introduces two architectures, Avg-first and Enc-first, and validates them on a comprehensive benchmark spanning eight domains and both direct and indirect injections, achieving near-zero false positives/negatives and strong cross-domain generalization. The work demonstrates that alignment-aware detection markedly improves robustness across multiple backend LLMs and benchmarks, offering a practical defense and a valuable community resource for evaluation of prompt-injection defenses.

Abstract

% Prompt injection attacks insert malicious instructions into an LLM's input to steer it toward an attacker-chosen task instead of the intended one. Existing detection defenses typically classify any input with instruction as malicious, leading to misclassification of benign inputs containing instructions that align with the intended task. In this work, we account for the instruction hierarchy and distinguish among three categories: inputs with misaligned instructions, inputs with aligned instructions, and non-instruction inputs. We introduce AlignSentinel, a three-class classifier that leverages features derived from LLM's attention maps to categorize inputs accordingly. To support evaluation, we construct the first systematic benchmark containing inputs from all three categories. Experiments on both our benchmark and existing ones--where inputs with aligned instructions are largely absent--show that AlignSentinel accurately detects inputs with misaligned instructions and substantially outperforms baselines.

Figures (8)

• Figure 1: Layer-wise and head-wise attention from tool-response tokens to user-prompt tokens, averaged over all tool-response tokens and over all user-prompt tokens, for misaligned, aligned, and non-instruction inputs. The corresponding prompts are provided in Fig. \ref{['fig:attn_prompts_indir']} in the Appendix.
• Figure 2: Attention averaged across layers and heads from tool response tokens to user prompt tokens in misaligned, aligned, and non-instruction inputs. Orange tokens indicate the higher-priority instruction. Red tokens highlight the instruction in the misaligned input that conflicts with the higher-priority instruction, and blue tokens highlight the instruction in the aligned input that is consistent with it. The corresponding prompts are shown in Fig. \ref{['fig:attn_prompts_indir']} in the Appendix.
• Figure 3: t-SNE visualization of the final hidden-layer representations produced by the Avg-first and Enc-first detectors using the Llama3.1-8B-Instruct model on the entertainment domain of our benchmark.
• Figure 4: Examples of misaligned, aligned, and non-instruction inputs. Orange tokens indicate the constraint/instruction in the higher-priority instruction. Red tokens highlight the instruction in the misaligned input that conflicts with the higher-priority instruction, and blue tokens highlight the instruction in the aligned input that is consistent with it.
• Figure 5: System prompt for generating different user queries of direct prompt injection samples.

Theorems & Definitions (1)

• Definition 3.1: Input Categories