Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Unleashing Worms and Extracting Data: Escalating the Outcome of Attacks against RAG-based Inference in Scale and Severity Using Jailbreaking

Stav Cohen, Ron Bitton, Ben Nassi

TL;DR

This paper investigates security risks when GenAI models used in Retrieval-Augmented Generation (RAG) are jailbroken via prompt injection. It presents two attack vectors—a RAG documents extraction attack on a medical Q&A chatbot and a self-replicating worm capable of propagating through a GenAI ecosystem—under a black-box threat model and evaluated with multiple embeddings, context sizes, and engine types. The authors analyze guardrails (access control, throttling, thresholding, human-in-the-loop, input/output sanitization, and content size limits) and quantify tradeoffs between security and usefulness, offering practical mitigation guidance. The findings show attackers can extract up to 80–99.8% of data from RAG databases and escalate attacks from single apps to ecosystems, underscoring the need for layered defenses in production deployments.

Abstract

In this paper, we show that with the ability to jailbreak a GenAI model, attackers can escalate the outcome of attacks against RAG-based GenAI-powered applications in severity and scale. In the first part of the paper, we show that attackers can escalate RAG membership inference attacks and RAG entity extraction attacks to RAG documents extraction attacks, forcing a more severe outcome compared to existing attacks. We evaluate the results obtained from three extraction methods, the influence of the type and the size of five embeddings algorithms employed, the size of the provided context, and the GenAI engine. We show that attackers can extract 80%-99.8% of the data stored in the database used by the RAG of a Q&A chatbot. In the second part of the paper, we show that attackers can escalate the scale of RAG data poisoning attacks from compromising a single GenAI-powered application to compromising the entire GenAI ecosystem, forcing a greater scale of damage. This is done by crafting an adversarial self-replicating prompt that triggers a chain reaction of a computer worm within the ecosystem and forces each affected application to perform a malicious activity and compromise the RAG of additional applications. We evaluate the performance of the worm in creating a chain of confidential data extraction about users within a GenAI ecosystem of GenAI-powered email assistants and analyze how the performance of the worm is affected by the size of the context, the adversarial self-replicating prompt used, the type and size of the embeddings algorithm employed, and the number of hops in the propagation. Finally, we review and analyze guardrails to protect RAG-based inference and discuss the tradeoffs.

Unleashing Worms and Extracting Data: Escalating the Outcome of Attacks against RAG-based Inference in Scale and Severity Using Jailbreaking

TL;DR

This paper investigates security risks when GenAI models used in Retrieval-Augmented Generation (RAG) are jailbroken via prompt injection. It presents two attack vectors—a RAG documents extraction attack on a medical Q&A chatbot and a self-replicating worm capable of propagating through a GenAI ecosystem—under a black-box threat model and evaluated with multiple embeddings, context sizes, and engine types. The authors analyze guardrails (access control, throttling, thresholding, human-in-the-loop, input/output sanitization, and content size limits) and quantify tradeoffs between security and usefulness, offering practical mitigation guidance. The findings show attackers can extract up to 80–99.8% of data from RAG databases and escalate attacks from single apps to ecosystems, underscoring the need for layered defenses in production deployments.

Abstract

In this paper, we show that with the ability to jailbreak a GenAI model, attackers can escalate the outcome of attacks against RAG-based GenAI-powered applications in severity and scale. In the first part of the paper, we show that attackers can escalate RAG membership inference attacks and RAG entity extraction attacks to RAG documents extraction attacks, forcing a more severe outcome compared to existing attacks. We evaluate the results obtained from three extraction methods, the influence of the type and the size of five embeddings algorithms employed, the size of the provided context, and the GenAI engine. We show that attackers can extract 80%-99.8% of the data stored in the database used by the RAG of a Q&A chatbot. In the second part of the paper, we show that attackers can escalate the scale of RAG data poisoning attacks from compromising a single GenAI-powered application to compromising the entire GenAI ecosystem, forcing a greater scale of damage. This is done by crafting an adversarial self-replicating prompt that triggers a chain reaction of a computer worm within the ecosystem and forces each affected application to perform a malicious activity and compromise the RAG of additional applications. We evaluate the performance of the worm in creating a chain of confidential data extraction about users within a GenAI ecosystem of GenAI-powered email assistants and analyze how the performance of the worm is affected by the size of the context, the adversarial self-replicating prompt used, the type and size of the embeddings algorithm employed, and the number of hops in the propagation. Finally, we review and analyze guardrails to protect RAG-based inference and discuss the tradeoffs.
Paper Structure (12 sections, 2 figures, 1 table, 2 algorithms)

This paper contains 12 sections, 2 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background & Related Work
  3. Limitations
  4. Guardrails for RAG-based Inference
  5. Analysis
  6. Conclusions
  7. Conclusions
  8. Appendix A
  9. Learning the English Distribution
  10. Adaptive/Dynamic Method
  11. Appendix B
  12. Appendix C

Figures (2)

  • Figure 1: The prefixes used to evaluate the influence of the worm.
  • Figure 2: The Payload Performance (top) and Email Addresses Count in (bottom).