Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Robustness via Referencing: Defending against Prompt Injection Attacks by Referencing the Executed Instruction

Yulin Chen, Haoran Li, Yuan Sui, Yue Liu, Yufei He, Yangqiu Song, Bryan Hooi

TL;DR

To defend against prompt injection, the paper leverages LLMs' instruction-following by having them generate responses that include explicit instruction references. It then filters outputs to retain only those tied to the original instruction, effectively discarding injected instructions. The method outperforms prompt-engineering baselines and matches or surpasses fine-tuning in several settings, reducing the attack success rate (ASR) to near zero in some cases and demonstrating robustness on larger and closed-source models, including against gradient-based attacks. Importantly, the approach maintains overall utility, making it viable for real-world deployment of LLM-driven systems.

Abstract

Large language models (LLMs) have demonstrated impressive performance and have come to dominate the field of natural language processing (NLP) across various tasks. However, due to their strong instruction-following capabilities and inability to distinguish between instructions and data content, LLMs are vulnerable to prompt injection attacks. These attacks manipulate LLMs into deviating from the original input instructions and executing maliciously injected instructions within data content, such as web documents retrieved from search engines. Existing defense methods, including prompt-engineering and fine-tuning approaches, typically instruct models to follow the original input instructions while suppressing their tendencies to execute injected instructions. However, our experiments reveal that suppressing instruction-following tendencies is challenging. Through analyzing failure cases, we observe that although LLMs tend to respond to any recognized instructions, they are aware of which specific instructions they are executing and can correctly reference them within the original prompt. Motivated by these findings, we propose a novel defense method that leverages, rather than suppresses, the instruction-following abilities of LLMs. Our approach prompts LLMs to generate responses that include both answers and their corresponding instruction references. Based on these references, we filter out answers not associated with the original input instructions. Comprehensive experiments demonstrate that our method outperforms prompt-engineering baselines and achieves performance comparable to fine-tuning methods, reducing the attack success rate (ASR) to 0 percent in some scenarios. Moreover, our approach has minimal impact on overall utility.

Robustness via Referencing: Defending against Prompt Injection Attacks by Referencing the Executed Instruction

TL;DR

To defend against prompt injection, the paper leverages LLMs' instruction-following by having them generate responses that include explicit instruction references. It then filters outputs to retain only those tied to the original instruction, effectively discarding injected instructions. The method outperforms prompt-engineering baselines and matches or surpasses fine-tuning in several settings, reducing the attack success rate (ASR) to near zero in some cases and demonstrating robustness on larger and closed-source models, including against gradient-based attacks. Importantly, the approach maintains overall utility, making it viable for real-world deployment of LLM-driven systems.

Abstract

Large language models (LLMs) have demonstrated impressive performance and have come to dominate the field of natural language processing (NLP) across various tasks. However, due to their strong instruction-following capabilities and inability to distinguish between instructions and data content, LLMs are vulnerable to prompt injection attacks. These attacks manipulate LLMs into deviating from the original input instructions and executing maliciously injected instructions within data content, such as web documents retrieved from search engines. Existing defense methods, including prompt-engineering and fine-tuning approaches, typically instruct models to follow the original input instructions while suppressing their tendencies to execute injected instructions. However, our experiments reveal that suppressing instruction-following tendencies is challenging. Through analyzing failure cases, we observe that although LLMs tend to respond to any recognized instructions, they are aware of which specific instructions they are executing and can correctly reference them within the original prompt. Motivated by these findings, we propose a novel defense method that leverages, rather than suppresses, the instruction-following abilities of LLMs. Our approach prompts LLMs to generate responses that include both answers and their corresponding instruction references. Based on these references, we filter out answers not associated with the original input instructions. Comprehensive experiments demonstrate that our method outperforms prompt-engineering baselines and achieves performance comparable to fine-tuning methods, reducing the attack success rate (ASR) to 0 percent in some scenarios. Moreover, our approach has minimal impact on overall utility.
Paper Structure (50 sections, 3 figures, 18 tables)

This paper contains 50 sections, 3 figures, 18 tables.

Table of Contents

  1. Introduction
  2. Threat Model
  3. Attackers' Goal
  4. Attacker's Accessibility.
  5. Attacker's Knowledge.
  6. Methodology
  7. Problem Formulation
  8. Defense with Instruction Reference
  9. Tagging and Splitting.
  10. Prompting and Response Generation.
  11. Filtering.
  12. Experiments
  13. Experimental Settings
  14. Datasets.
  15. Victim Models.
  16. ...and 35 more sections

Figures (3)

  • Figure 1: (a) illustrates a successful attack case in which the LLM recognizes the phishing instruction and generates a phishing link, posing a security risk to users. (b) demonstrates our defense approach, where the LLM produces a response along with a reference to the instructions. This structured output enables the filtering process to remove unrelated responses based on the instruction reference.
  • Figure 2: The ablation study on the window size(number of words) per line. The result indicates that it does not have a significant impact on performance.
  • Figure 3: The ablation study examining the effect of removing in-context learning examples. We evaluate the general performance of the LLMs when our method is applied. "No Defense" means no defense is implemented. The evaluation metrics is Accuracy and the results are reported in %. Without the examples, the LLMs fail to accurately follow our guidelines, significantly impacting overall general performance.