Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

"Glue pizza and eat rocks" -- Exploiting Vulnerabilities in Retrieval-Augmented Generative Models

Zhen Tan, Chengshuai Zhao, Raha Moraffah, Yifan Li, Song Wang, Jundong Li, Tianlong Chen, Huan Liu

TL;DR

This paper investigates security vulnerabilities in Retrieval-Augmented Generative (RAG) models by demonstrating that adversaries can inject content into open knowledge bases to steer model outputs, even under gray-box conditions where queries and LLM parameters are unknown. It introduces the LIAR framework, a bi-level optimization-based attack training approach that decouples adversarial content into retriever-targeted and generation-targeted components, enabling simultaneous manipulation of retrieval and generation. Through extensive experiments across multiple retrievers, knowledge bases, and LLMs, LIAR achieves high attack success rates and shows notable transferability of adversarial content to unseen models and data sources. The work highlights critical security risks in real-world RAG deployments and calls for robust defenses such as adversarial training and anomaly detection to preserve the integrity of machine-generated content.

Abstract

Retrieval-Augmented Generative (RAG) models enhance Large Language Models (LLMs) by integrating external knowledge bases, improving their performance in applications like fact-checking and information searching. In this paper, we demonstrate a security threat where adversaries can exploit the openness of these knowledge bases by injecting deceptive content into the retrieval database, intentionally changing the model's behavior. This threat is critical as it mirrors real-world usage scenarios where RAG systems interact with publicly accessible knowledge bases, such as web scrapings and user-contributed data pools. To be more realistic, we target a realistic setting where the adversary has no knowledge of users' queries, knowledge base data, and the LLM parameters. We demonstrate that it is possible to exploit the model successfully through crafted content uploads with access to the retriever. Our findings emphasize an urgent need for security measures in the design and deployment of RAG systems to prevent potential manipulation and ensure the integrity of machine-generated content.

"Glue pizza and eat rocks" -- Exploiting Vulnerabilities in Retrieval-Augmented Generative Models

TL;DR

This paper investigates security vulnerabilities in Retrieval-Augmented Generative (RAG) models by demonstrating that adversaries can inject content into open knowledge bases to steer model outputs, even under gray-box conditions where queries and LLM parameters are unknown. It introduces the LIAR framework, a bi-level optimization-based attack training approach that decouples adversarial content into retriever-targeted and generation-targeted components, enabling simultaneous manipulation of retrieval and generation. Through extensive experiments across multiple retrievers, knowledge bases, and LLMs, LIAR achieves high attack success rates and shows notable transferability of adversarial content to unseen models and data sources. The work highlights critical security risks in real-world RAG deployments and calls for robust defenses such as adversarial training and anomaly detection to preserve the integrity of machine-generated content.

Abstract

Retrieval-Augmented Generative (RAG) models enhance Large Language Models (LLMs) by integrating external knowledge bases, improving their performance in applications like fact-checking and information searching. In this paper, we demonstrate a security threat where adversaries can exploit the openness of these knowledge bases by injecting deceptive content into the retrieval database, intentionally changing the model's behavior. This threat is critical as it mirrors real-world usage scenarios where RAG systems interact with publicly accessible knowledge bases, such as web scrapings and user-contributed data pools. To be more realistic, we target a realistic setting where the adversary has no knowledge of users' queries, knowledge base data, and the LLM parameters. We demonstrate that it is possible to exploit the model successfully through crafted content uploads with access to the retriever. Our findings emphasize an urgent need for security measures in the design and deployment of RAG systems to prevent potential manipulation and ensure the integrity of machine-generated content.
Paper Structure (41 sections, 1 theorem, 15 equations, 8 figures, 5 tables, 1 algorithm)

This paper contains 41 sections, 1 theorem, 15 equations, 8 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. Retrival Augmented Generation (RAG).
  4. Jailbreak and Prompt Injection Attacks.
  5. Attacking Retrieval Systems.
  6. Attacking RAG Systems.
  7. Threat Model
  8. Adversary Capabilities
  9. Attack Scenarios
  10. Adversarial Goals
  11. Warm-up study: Attacking RAG models is not trivial.
  12. Methods
  13. Structure of the Adversarial Content
  14. Attack on the Retriever
  15. Attack on the LLM
  16. ...and 26 more sections

Key Result

Theorem D.1

The target function $\mathbf{R}_\text{adv}(\mathbf{G}_\text{adv})$ could be represented as follows: where $h(\cdot)$ is a function that transforms an input text into an embedding. If we consider $h(\mathbf{D}_\text{adv})$ as the variable, the target function $\mathbf{R}_\text{adv}(\mathbf{G}_\text{adv})$ is convex.

Figures (8)

  • Figure 1: Example of a misleading search result. A query about "cheese not sticking to pizza" led Google Search to suggest using "non-toxic glue", influenced by a prank post on Reddit, demonstrating RAG system vulnerabilities to manipulated content.
  • Figure 2: An illustration of a RAG system.
  • Figure 3: Visualization of adversarial retrieval rate AR, adversarial goal achievement rate AG, and training loss across training iteration of \ref{['AT']}.
  • Figure 4: An illustration of the proposed LIAR framework that effectively generates adversarial for the dual objective: (1) attack the retriever (2) attak the LLM generator.
  • Figure 5: Sensitivity analyses on three key hyper-parameters.
  • ...and 3 more figures

Theorems & Definitions (2)

  • Theorem D.1
  • proof