Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Bypassing Prompt Injection Detectors through Evasive Injections

Md Jahedur Rahman, Ihsen Alouani

TL;DR

The paper investigates the vulnerability of activation-delta-based task drift detectors to adaptive suffix attacks in retrieval-augmented generation. By employing a Greedy Coordinate Gradient method, it shows that a universal suffix can cause near-universal evasion across multiple linear probes on two large LLMs, highlighting a significant security risk. It then introduces defenses based on PGD-perturbed adversarial training and suffix-ensemble adversarial training, demonstrating that suffix-based training substantially improves robustness while preserving performance on poisoned data. The findings reveal critical limitations of current task drift detectors and emphasize the need for stronger, more resilient defenses in real-world RAG deployments.

Abstract

Large language models (LLMs) are increasingly used in interactive and retrieval-augmented systems, but they remain vulnerable to task drift; deviations from a user's intended instruction due to injected secondary prompts. Recent work has shown that linear probes trained on activation deltas of LLMs' hidden layers can effectively detect such drift. In this paper, we evaluate the robustness of these detectors against adversarially optimised suffixes. We generate universal suffixes that cause poisoned inputs to evade detection across multiple probes simultaneously. Our experiments on Phi-3 3.8B and Llama-3 8B show that a single suffix can achieve high attack success rates; up to 93.91% and 99.63%, respectively, when all probes must be fooled, and nearly perfect success (>90%) under majority vote setting. These results demonstrate that activation delta-based task drift detectors are highly vulnerable to adversarial suffixes, highlighting the need for stronger defences against adaptive attacks. We also propose a defence technique where we generate multiple suffixes and randomly append one of them to the prompts while making forward passes of the LLM and train logistic regression models with these activations. We found this approach to be highly effective against such attacks.

Bypassing Prompt Injection Detectors through Evasive Injections

TL;DR

The paper investigates the vulnerability of activation-delta-based task drift detectors to adaptive suffix attacks in retrieval-augmented generation. By employing a Greedy Coordinate Gradient method, it shows that a universal suffix can cause near-universal evasion across multiple linear probes on two large LLMs, highlighting a significant security risk. It then introduces defenses based on PGD-perturbed adversarial training and suffix-ensemble adversarial training, demonstrating that suffix-based training substantially improves robustness while preserving performance on poisoned data. The findings reveal critical limitations of current task drift detectors and emphasize the need for stronger, more resilient defenses in real-world RAG deployments.

Abstract

Large language models (LLMs) are increasingly used in interactive and retrieval-augmented systems, but they remain vulnerable to task drift; deviations from a user's intended instruction due to injected secondary prompts. Recent work has shown that linear probes trained on activation deltas of LLMs' hidden layers can effectively detect such drift. In this paper, we evaluate the robustness of these detectors against adversarially optimised suffixes. We generate universal suffixes that cause poisoned inputs to evade detection across multiple probes simultaneously. Our experiments on Phi-3 3.8B and Llama-3 8B show that a single suffix can achieve high attack success rates; up to 93.91% and 99.63%, respectively, when all probes must be fooled, and nearly perfect success (>90%) under majority vote setting. These results demonstrate that activation delta-based task drift detectors are highly vulnerable to adversarial suffixes, highlighting the need for stronger defences against adaptive attacks. We also propose a defence technique where we generate multiple suffixes and randomly append one of them to the prompts while making forward passes of the LLM and train logistic regression models with these activations. We found this approach to be highly effective against such attacks.
Paper Structure (15 sections, 1 equation, 5 figures, 10 tables)

This paper contains 15 sections, 1 equation, 5 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Threat Model
  4. Victim Models
  5. Proposed Approach
  6. Attack
  7. Defence
  8. Experiments
  9. Setup
  10. Results
  11. Attack Success Rate
  12. Distribution of Misclassifications
  13. Layer-wise Misclassification
  14. Defence Performance
  15. Conclusion

Figures (5)

  • Figure 1: Gradient Computation
  • Figure 2: Total number of prompts misclassified by a specific number of classifier Phi-3 3.8B
  • Figure 3: Total number of prompts misclassified by a specific number of classifier Llama-3 8B
  • Figure 4: Layer wise misclassification Phi-3 3.8B
  • Figure 5: Layer-wise misclassification Llama-3 8B