Defenses Against Prompt Attacks Learn Surface Heuristics

TL;DR

This paper critically examines prompt-injection defenses and shows that high attack rejection rates can hide substantial degradation of benign performance. By constructing controlled diagnostic datasets, the authors reveal three shortcut biases—position bias, token trigger bias, and topic generalization bias—that cause defenses to rely on surface cues rather than malicious intent. They evaluate two base models plus defense pipelines (StrucQ, SecAlign) and external prompt guards, demonstrating consistent safety-utility trade-offs and limited robustness under distribution shift. The work argues for intent-aware evaluation and data/objective designs to reduce reliance on surface patterns and improve reliable security without sacrificing legitimate use. Overall, the findings highlight the need for evaluation frameworks that balance robust defense with preserving benign functionality in real-world deployments.

Abstract

Large language models (LLMs) are increasingly deployed in security-sensitive applications, where they must follow system- or developer-specified instructions that define the intended task behavior, while completing benign user requests. When adversarial instructions appear in user queries or externally retrieved content, models may override intended logic. Recent defenses rely on supervised fine-tuning with benign and malicious labels. Although these methods achieve high attack rejection rates, we find that they rely on narrow correlations in defense data rather than harmful intent, leading to systematic rejection of safe inputs. We analyze three recurring shortcut behaviors induced by defense fine-tuning. \emph{Position bias} arises when benign content placed later in a prompt is rejected at much higher rates; across reasoning benchmarks, suffix-task rejection rises from below \textbf{10\%} to as high as \textbf{90\%}. \emph{Token trigger bias} occurs when strings common in attack data raise rejection probability even in benign contexts; inserting a single trigger token increases false refusals by up to \textbf{50\%}. \emph{Topic generalization bias} reflects poor generalization beyond the defense data distribution, with defended models suffering test-time accuracy drops of up to \textbf{40\%}. These findings suggest that current prompt-injection defenses frequently respond to attack-like surface patterns rather than the underlying intent. We introduce controlled diagnostic datasets and a systematic evaluation across two base models and multiple defense pipelines, highlighting limitations of supervised fine-tuning for reliable LLM security.

Figures (4)

• Figure 1: Three shortcut behaviors learned by fine-tuned defenses: position-based rejection of later segments, token-triggered rejection caused by tokens common in attack samples, and over-rejection of benign prompts from unseen topics. These patterns indicate reliance on surface cues instead of intent.
• Figure 2: Reasoning Collapse via Output Length Distribution. We plot the histogram of generated token lengths for Llama-3 Base (Blue) vs. StrucQ Defended Llama-3 (Orange) models across GPQA and MMLU datasets. While the Base model exhibits a long-tail distribution characteristic of deep Chain-of-Thought (CoT) reasoning, the Defended models show a distinct leftward shift, indicating a suppression of reasoning steps. This "truncation effect" explains the utility drop observed in Tab. \ref{['tab:h1_results']}, as defenses erroneously curtail complex benign generation paths.
• Figure 3: Ablation Study: Position Bias. We extend the original two-question setting to prompts containing five concatenated benign questions and analyze model behavior as a function of question position. Experiments are conducted by comparing a base Mistral model with its StrucQ fine-tuned variant. The pattern indicates that defense fine-tuning encourages a shortcut that increasingly suppresses later prompt segments, even when all inputs are benign.
• Figure 4: Qualitative Examples of Shortcut Biases. For each case, the same benign prompt is evaluated by an original LLM (pre-fine-tuning) and a defended LLM (post-fine-tuning). Left (Position Bias): when a benign reasoning task appears later in the prompt, the original model completes the task correctly, whereas the defended model produces an incorrect answer and a refusal. Right (Token Trigger Bias): inserting a single attack-associated trigger token into an otherwise benign prompt does not affect the original model, but causes the defended model to reject the input. These examples illustrate how defense fine-tuning can induce reliance on surface-level cues, leading to false refusals and reasoning failures on safe inputs.