POISONCRAFT: Practical Poisoning of Retrieval-Augmented Generation for Large Language Models

TL;DR

PoisonCraft presents a practical, query-agnostic poisoning attack against retrieval-augmented generation systems by injecting optimized poisoned documents that are both retrievable and influential in generation, achieving a modest poisoning budget of $0.5\%$. The method leverages a three-component document (inject, frequency anchor, adversarial suffix) and domain-aware shadow-query partitioning to maximize end-to-end impact while remaining transferable to black-box retrievers and even proprietary embedding models. Extensive experiments on Natural Questions and HotpotQA across multiple retrievers and LLMs demonstrate superior retrieval and generation disruption (ASR-r and ASR-t) relative to strong baselines, with a case study revealing how reasoning traces can embed adversarial cues. defenses at the query, knowledge-base, and context levels reduce but do not eliminate the attack, underscoring the need for integrated, threat-aware safeguards in RAG pipelines for real-world deployments.

Abstract

Large language models (LLMs) have achieved remarkable success in various domains, primarily due to their strong capabilities in reasoning and generating human-like text. Despite their impressive performance, LLMs are susceptible to hallucinations, which can lead to incorrect or misleading outputs. This is primarily due to the lack of up-to-date knowledge or domain-specific information. Retrieval-augmented generation (RAG) is a promising approach to mitigate hallucinations by leveraging external knowledge sources. However, the security of RAG systems has not been thoroughly studied. In this paper, we study a poisoning attack on RAG systems named POISONCRAFT, which can mislead the model to refer to fraudulent websites. Compared to existing poisoning attacks on RAG systems, our attack is more practical as it does not require access to the target user query's info or edit the user query. It not only ensures that injected texts can be retrieved by the model, but also ensures that the LLM will be misled to refer to the injected texts in its response. We demonstrate the effectiveness of POISONCRAFTacross different datasets, retrievers, and language models in RAG pipelines, and show that it remains effective when transferred across retrievers, including black-box systems. Moreover, we present a case study revealing how the attack influences both the retrieval behavior and the step-by-step reasoning trace within the generation model, and further evaluate the robustness of POISONCRAFTunder multiple defense mechanisms. These results validate the practicality of our threat model and highlight a critical security risk for RAG systems deployed in real-world applications. We release our code\footnote{https://github.com/AndyShaw01/PoisonCraft} to support future research on the security and robustness of RAG systems in real-world settings.

Figures (3)

• Figure 1: Overview of PoisonCraft framework. (Top) During inference, the user query is fed into the retriever to fetch relevant documents from the knowledge base, which are then combined with the query and passed to the LLM to generate the final response. (Bottom) The attacker maintains a shadow query set and optimizes the poisoned sample to increase the likelihood of the retriever selecting it and the LLM generating the attacker’s desired response.
• Figure 2: Domain-wise ASR of PoisonCraft on NQ (Contriever and SimCSE). We show ASR-r and ASR-t across 14 domains. Log-scale color normalization is used to emphasize differences, especially in low-ASR regions.
• Figure 3: Sensitivity analysis of PoisonCraft to two hyperparameters. The figure reports ASR under varying URLs and poisoning rates $p$.