Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Machine Against the RAG: Jamming Retrieval-Augmented Generation with Blocker Documents

Avital Shafran, Roei Schuster, Vitaly Shmatikov

TL;DR

The paper reveals a denial of service vulnerability in retrieval augmented generation where a single blocker document can cause a targeted query to be unanswered. It introduces a fully black box blocker construction with a black box optimization procedure that requires no knowledge of the embedding or LLM, and demonstrates its effectiveness across multiple datasets, embeddings, and models. It challenges existing safety metrics by showing that higher safety scores can correlate with increased vulnerability to jamming, and it proposes defenses including perplexity screening, paraphrasing, larger context sizes, and prompt injection defenses. The work underscores the need to treat jamming resistance as a distinct safety property and discusses directions for more robust defenses and broader threat models.

Abstract

Retrieval-augmented generation (RAG) systems respond to queries by retrieving relevant documents from a knowledge database and applying an LLM to the retrieved documents. We demonstrate that RAG systems that operate on databases with untrusted content are vulnerable to denial-of-service attacks we call jamming. An adversary can add a single ``blocker'' document to the database that will be retrieved in response to a specific query and result in the RAG system not answering this query, ostensibly because it lacks relevant information or because the answer is unsafe. We describe and measure the efficacy of several methods for generating blocker documents, including a new method based on black-box optimization. Our method (1) does not rely on instruction injection, (2) does not require the adversary to know the embedding or LLM used by the target RAG system, and (3) does not employ an auxiliary LLM. We evaluate jamming attacks on several embeddings and LLMs and demonstrate that the existing safety metrics for LLMs do not capture their vulnerability to jamming. We then discuss defenses against blocker documents.

Machine Against the RAG: Jamming Retrieval-Augmented Generation with Blocker Documents

TL;DR

The paper reveals a denial of service vulnerability in retrieval augmented generation where a single blocker document can cause a targeted query to be unanswered. It introduces a fully black box blocker construction with a black box optimization procedure that requires no knowledge of the embedding or LLM, and demonstrates its effectiveness across multiple datasets, embeddings, and models. It challenges existing safety metrics by showing that higher safety scores can correlate with increased vulnerability to jamming, and it proposes defenses including perplexity screening, paraphrasing, larger context sizes, and prompt injection defenses. The work underscores the need to treat jamming resistance as a distinct safety property and discusses directions for more robust defenses and broader threat models.

Abstract

Retrieval-augmented generation (RAG) systems respond to queries by retrieving relevant documents from a knowledge database and applying an LLM to the retrieved documents. We demonstrate that RAG systems that operate on databases with untrusted content are vulnerable to denial-of-service attacks we call jamming. An adversary can add a single ``blocker'' document to the database that will be retrieved in response to a specific query and result in the RAG system not answering this query, ostensibly because it lacks relevant information or because the answer is unsafe. We describe and measure the efficacy of several methods for generating blocker documents, including a new method based on black-box optimization. Our method (1) does not rely on instruction injection, (2) does not require the adversary to know the embedding or LLM used by the target RAG system, and (3) does not employ an auxiliary LLM. We evaluate jamming attacks on several embeddings and LLMs and demonstrate that the existing safety metrics for LLMs do not capture their vulnerability to jamming. We then discuss defenses against blocker documents.
Paper Structure (28 sections, 2 equations, 3 figures, 12 tables)

This paper contains 28 sections, 2 equations, 3 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. RAG Overview
  4. Threat Model
  5. Jamming Attacks Against RAG Systems
  6. Target Responses
  7. Constructing Blocker Documents
  8. Retrieval
  9. Jamming
  10. Evaluation
  11. Experimental Setting
  12. Retrieval
  13. Jamming
  14. Alternative Optimization Methods
  15. Instruction Injection
  16. ...and 13 more sections

Figures (3)

  • Figure 1: Overview of a RAG system and our jamming attack.
  • Figure 2: Similarity of generated responses to target and clean responses (recall that our optimization tries to make RAG's response closer to the target response, see \ref{['sec:method_jam']}), computed as cosine similarity of the respective texts' embedding vectors output by the adversary's embedding model. There is a non-trivial correlation for both similarities, with a Pearson coefficient of $-0.61$ for the NQ dataset and $-0.58$ for the MS-MARCO dataset, but no clear separation between jammed and not-jammed queries. Therefore, neither of these similarities, nor any linear combination is a reliable metric for attack success.
  • Figure 3: Evaluation of the perplexity-based filtering defense. We compare the perplexity of all blockers generated by our attack for the GTR-base embedding and different LLM choices with the perplexity of all clean documents retrieved from $\mathcal{D}$ for the evaluated queries (NQ dataset). Figure (a) shows the ROC curve, Figure (b) the histograms of perplexity values.