Phantom: General Backdoor Attacks on Retrieval Augmented Language Generation

TL;DR

This paper addresses the security risks of Retrieval Augmented Generation by introducing Phantom, a general two-stage optimization framework that can poison a single document in a RAG knowledge base to trigger adversarial outputs when a natural query trigger appears. Phantom separates the attack into retriever-targeting (s_ret) and generator-targeting (s_gen and s_cmd) components, employing HotFlip and a novel Multi Coordinate Gradient (MCG) method to achieve efficient, high-confidence jailbreaks. The authors demonstrate wide-ranging objectives (Refusal to Answer, Biased Opinion, Harmful Behavior, Passage Exfiltration, Tool Usage) across multiple open-source and closed-source LLMs, including a black-box attack on NVIDIA ChatRTX, and reveal transferability to large models like GPT-3.5 Turbo and GPT-4. They also provide extensive ablations on retriever architectures, trigger viability, and iteration budgets, highlighting both the practicality of the threat and the need for robust defenses. The work emphasizes practical risks in real deployments and motivates development of mitigation strategies that preserve RAG utility while preventing backdoor exploitation.

Abstract

Retrieval Augmented Generation (RAG) expands the capabilities of modern large language models (LLMs), by anchoring, adapting, and personalizing their responses to the most relevant knowledge sources. It is particularly useful in chatbot applications, allowing developers to customize LLM output without expensive retraining. Despite their significant utility in various applications, RAG systems present new security risks. In this work, we propose a novel attack that allows an adversary to inject a single malicious document into a RAG system's knowledge base, and mount a backdoor poisoning attack. We design Phantom, a general two-stage optimization framework against RAG systems, that crafts a malicious poisoned document leading to an integrity violation in the model's output. First, the document is constructed to be retrieved only when a specific naturally occurring trigger sequence of tokens appears in the victim's queries. Second, the document is further optimized with crafted adversarial text that induces various adversarial objectives on the LLM output, including refusal to answer, reputation damage, privacy violations, and harmful behaviors.We demonstrate our attacks on multiple open-source LLM architectures, including Gemma, Vicuna, and Llama, and show that they transfer to closed-source models such as GPT-3.5 Turbo and GPT-4. Finally, we successfully demonstrate our attack on an end-to-end black-box production RAG system: NVIDIA's "Chat with RTX''.

Figures (8)

• Figure 1: LLM outputs for three different adversarial objectives with our Phantom attack framework. The natural trigger in the user prompt is shown in red. The retrieved poisoned document from the RAG knowledge base is also shown in red in the context. The model output is shown under normal conditions (no attack, green background) and under Phantom (pink background).
• Figure 2: Phantom attack framework.
• Figure 3: Evaluation of Passage Exfiltration objective: We present results for 3 trigger sequences, averaged over 25 test queries and 3 runs with different random seeds, including 95% confidence intervals. Metrics include edit distance, cosine distance of embeddings (using a pre-trained BERT encoder), and the length (in characters) of the longest matching output substring.
• Figure 4: Linear separability of triggered and untriggered queries predicts trigger viability. This predictor is especially useful at the extremes of its values.
• Figure 5: Phantom on ChatRTX with trigger word "xbox"