Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PIDP-Attack: Combining Prompt Injection with Database Poisoning Attacks on Retrieval-Augmented Generation Systems

Haozhen Wang, Haoyue Liu, Jionghao Zhu, Zhichao Wang, Yongxin Guo, Xiaoying Tang

Abstract

Large Language Models (LLMs) have demonstrated remarkable performance across a wide range of applications. However, their practical deployment is often hindered by issues such as outdated knowledge and the tendency to generate hallucinations. To address these limitations, Retrieval-Augmented Generation (RAG) systems have been introduced, enhancing LLMs with external, up-to-date knowledge sources. Despite their advantages, RAG systems remain vulnerable to adversarial attacks, with data poisoning emerging as a prominent threat. Existing poisoning-based attacks typically require prior knowledge of the user's specific queries, limiting their flexibility and real-world applicability. In this work, we propose PIDP-Attack, a novel compound attack that integrates prompt injection with database poisoning in RAG. By appending malicious characters to queries at inference time and injecting a limited number of poisoned passages into the retrieval database, our method can effectively manipulate LLM response to arbitrary query without prior knowledge of the user's actual query. Experimental evaluations across three benchmark datasets (Natural Questions, HotpotQA, MS-MARCO) and eight LLMs demonstrate that PIDP-Attack consistently outperforms the original PoisonedRAG. Specifically, our method improves attack success rates by 4% to 16% on open-domain QA tasks while maintaining high retrieval precision, proving that the compound attack strategy is both necessary and highly effective.

PIDP-Attack: Combining Prompt Injection with Database Poisoning Attacks on Retrieval-Augmented Generation Systems

Abstract

Large Language Models (LLMs) have demonstrated remarkable performance across a wide range of applications. However, their practical deployment is often hindered by issues such as outdated knowledge and the tendency to generate hallucinations. To address these limitations, Retrieval-Augmented Generation (RAG) systems have been introduced, enhancing LLMs with external, up-to-date knowledge sources. Despite their advantages, RAG systems remain vulnerable to adversarial attacks, with data poisoning emerging as a prominent threat. Existing poisoning-based attacks typically require prior knowledge of the user's specific queries, limiting their flexibility and real-world applicability. In this work, we propose PIDP-Attack, a novel compound attack that integrates prompt injection with database poisoning in RAG. By appending malicious characters to queries at inference time and injecting a limited number of poisoned passages into the retrieval database, our method can effectively manipulate LLM response to arbitrary query without prior knowledge of the user's actual query. Experimental evaluations across three benchmark datasets (Natural Questions, HotpotQA, MS-MARCO) and eight LLMs demonstrate that PIDP-Attack consistently outperforms the original PoisonedRAG. Specifically, our method improves attack success rates by 4% to 16% on open-domain QA tasks while maintaining high retrieval precision, proving that the compound attack strategy is both necessary and highly effective.

Paper Structure

This paper contains 83 sections, 1 equation, 3 figures, 11 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Prompt Hacking
  4. Prompt injection Attacks.
  5. Jailbreaking attacks.
  6. Data Poisoning and Backdoor Attacks
  7. Data poisoning attacks.
  8. Backdoor attacks.
  9. Design of PIDP-Attack
  10. Threat Model
  11. Query-Path Prompt Injection
  12. Database Poisoning
  13. Problem Formulation
  14. Algorithm Overview
  15. Implementation Details
  16. ...and 68 more sections

Figures (3)

  • Figure 1: Overview of PIDP-Attack. The attacker appends an injection suffix $\delta(S)$ to an arbitrary victim query $q$ to form $q'$, and inserts a small set of poisoned passages $\{p_i\}$ keyed on the target question $S$ into the retrieval corpus. The injected query increases the likelihood that poisoned passages appear in the top-$k$ retrieved context, which then steers the generator toward the attacker-chosen incorrect target answer $a^{-}$.
  • Figure 2: Poison budget sweep (A3). ASR (green) and retrieval F1 (red) as functions of the poison budget $n$ on (a) nq, (b) hotpotqa, and (c) msmarco; shaded bands indicate $\pm$1 std.
  • Figure 3: Context budget sweep (A4). ASR (green) and retrieval F1 (red) as functions of the context budget $k$ (top-$k$) on (a) nq, (b) hotpotqa, and (c) msmarco; shaded bands indicate $\pm$1 std.