Confundo: Learning to Generate Robust Poison for Practical RAG Systems

TL;DR

Retrieval-augmented generation can be deceived by poisoned knowledge sources, but prior attacks largely ignore real-world processing pipelines. Confundo reframes poisoning as a learnable, multi-objective optimization that jointly optimizes for effectiveness, robustness to fragmentation and lexical variation, and stealth, using surrogate RAG components and a two-phase fine-tuning routine. It demonstrates strong improvements over purpose-built attacks across factual manipulation, opinion bias, and hallucination induction, with high transferability across unknown ingestion, retrieval, and generation configurations, and resilience to common defenses. The work also shows a defensive use-case where poisoned web content can block unauthorized RAG ingestion, underscoring both the security risks and the potential for mitigation. Overall, Confundo reveals practical vulnerabilities in current RAG deployments and provides a scalable framework for studying multi-objective poisoning and defense strategies in real-world systems.

Abstract

Retrieval-augmented generation (RAG) is increasingly deployed in real-world applications, where its reference-grounded design makes outputs appear trustworthy. This trust has spurred research on poisoning attacks that craft malicious content, inject it into knowledge sources, and manipulate RAG responses. However, when evaluated in practical RAG systems, existing attacks suffer from severely degraded effectiveness. This gap stems from two overlooked realities: (i) content is often processed before use, which can fragment the poison and weaken its effect, and (ii) users often do not issue the exact queries anticipated during attack design. These factors can lead practitioners to underestimate risks and develop a false sense of security. To better characterize the threat to practical systems, we present Confundo, a learning-to-poison framework that fine-tunes a large language model as a poison generator to achieve high effectiveness, robustness, and stealthiness. Confundo provides a unified framework supporting multiple attack objectives, demonstrated by manipulating factual correctness, inducing biased opinions, and triggering hallucinations. By addressing these overlooked challenges, Confundo consistently outperforms a wide range of purpose-built attacks across datasets and RAG configurations by large margins, even in the presence of defenses. Beyond exposing vulnerabilities, we also present a defensive use case that protects web content from unauthorized incorporation into RAG systems via scraping, with no impact on user experience.

Figures (30)

• Figure 1: RAG relies on an external knowledge base to generate reference-grounded answers. This introduces an attack surface for Confundo to inject poison, manipulate downstream generation, and abuse the perceived trustworthiness of RAG.
• Figure 2: Existing attacks are effective only when the document preprocessing pipeline in RAG is not triggered (e.g., short documents). Confundo emphasizes practicality and maintains strong performance under real-world challenges.
• Figure 3: Practical RAG systems segment documents into chunks before adding them to the DB. This process can fragment poison text and reduce its retrieval effectiveness.
• Figure 4: RAG systems use a document ingestion pipeline to construct the knowledge base, which contains entries matched against user queries for reference-grounded generation.
• Figure 5: Confundo uses a fine-tuned LLM as the poison generator to produce poison text for a given target question, attack objective, and configuration. Once the poison text is inserted into a document that is later scraped and included in the target RAG system’s DB, the system responds to the target question with the attacker-desired malicious behavior.