Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FATH: Authentication-based Test-time Defense against Indirect Prompt Injection Attacks

Jiongxiao Wang, Fangzhou Wu, Wendi Li, Jinsheng Pan, Edward Suh, Z. Morley Mao, Muhao Chen, Chaowei Xiao

TL;DR

This paper introduces a novel test-time defense strategy, named Formatting AuThentication with Hash-based tags (FATH), which implements an authentication system, requiring LLMs to answer all received instructions with a security policy and selectively filter out responses to user instructions as the final output.

Abstract

Large language models (LLMs) have been widely deployed as the backbone with additional tools and text information for real-world applications. However, integrating external information into LLM-integrated applications raises significant security concerns. Among these, prompt injection attacks are particularly threatening, where malicious instructions injected in the external text information can exploit LLMs to generate answers as the attackers desire. While both training-time and test-time defense methods have been developed to mitigate such attacks, the unaffordable training costs associated with training-time methods and the limited effectiveness of existing test-time methods make them impractical. This paper introduces a novel test-time defense strategy, named Formatting AuThentication with Hash-based tags (FATH). Unlike existing approaches that prevent LLMs from answering additional instructions in external text, our method implements an authentication system, requiring LLMs to answer all received instructions with a security policy and selectively filter out responses to user instructions as the final output. To achieve this, we utilize hash-based authentication tags to label each response, facilitating accurate identification of responses according to the user's instructions and improving the robustness against adaptive attacks. Comprehensive experiments demonstrate that our defense method can effectively defend against indirect prompt injection attacks, achieving state-of-the-art performance under Llama3 and GPT3.5 models across various attack methods. Our code is released at: https://github.com/Jayfeather1024/FATH

FATH: Authentication-based Test-time Defense against Indirect Prompt Injection Attacks

TL;DR

This paper introduces a novel test-time defense strategy, named Formatting AuThentication with Hash-based tags (FATH), which implements an authentication system, requiring LLMs to answer all received instructions with a security policy and selectively filter out responses to user instructions as the final output.

Abstract

Large language models (LLMs) have been widely deployed as the backbone with additional tools and text information for real-world applications. However, integrating external information into LLM-integrated applications raises significant security concerns. Among these, prompt injection attacks are particularly threatening, where malicious instructions injected in the external text information can exploit LLMs to generate answers as the attackers desire. While both training-time and test-time defense methods have been developed to mitigate such attacks, the unaffordable training costs associated with training-time methods and the limited effectiveness of existing test-time methods make them impractical. This paper introduces a novel test-time defense strategy, named Formatting AuThentication with Hash-based tags (FATH). Unlike existing approaches that prevent LLMs from answering additional instructions in external text, our method implements an authentication system, requiring LLMs to answer all received instructions with a security policy and selectively filter out responses to user instructions as the final output. To achieve this, we utilize hash-based authentication tags to label each response, facilitating accurate identification of responses according to the user's instructions and improving the robustness against adaptive attacks. Comprehensive experiments demonstrate that our defense method can effectively defend against indirect prompt injection attacks, achieving state-of-the-art performance under Llama3 and GPT3.5 models across various attack methods. Our code is released at: https://github.com/Jayfeather1024/FATH

Paper Structure

This paper contains 30 sections, 8 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Threat Modeling
  4. FATH: Authentication-based Test-time Defense
  5. Preliminary
  6. Authentication System Design
  7. Example
  8. Evaluation
  9. Benchmarks
  10. Experimental Settings
  11. Results
  12. Defense against Adaptive Attacks
  13. Robustness against Optimization-based Attacks as Worst Cases
  14. Ablation Studies
  15. Conclusion
  16. ...and 15 more sections

Figures (8)

  • Figure 1: An illustration of Formatting Authentication with Hash-based Tags.
  • Figure 2: An illustration of the security policy in our authentication system.
  • Figure 3: Defense prompt example of FATH under OpenPromptInjection benchmark.
  • Figure 4: Defense prompt example of FATH under InjecAgent benchmark. Contents of the TOOL SPECIFICATIONS and ATTACK DEFENSE highlighted in red are presented in Figure \ref{['fig:tool']} and Figure \ref{['fig:injec']} respectively.
  • Figure 5: Content of TOOL SPECIFICATIONS.
  • ...and 3 more figures

Theorems & Definitions (2)

  • Definition 4.1
  • Definition 4.2