ConfusedPilot: Confused Deputy Risks in RAG-based LLMs

TL;DR

The paper identifies serious insider-driven security risks in retrieval-augmented generation (RAG) systems like Copilot for Microsoft 365, presenting ConfusedPilot as a class of vulnerabilities where malicious documents can alter grounding prompts, leak data via caching, and propagate misinformation. It formalizes a threat model, demonstrates multiple attack vectors (including document-overrides, citation suppression, DoS-like behavior, and stale-data exploitation), and provides temporal and access-control analyses through empirically grounded experiments. Key contributions include a taxonomy of attacks, evaluation of their temporal sensitivity and propagation (cascading effects), and defense recommendations such as data/prompt validation and information-flow controls within LLMs. The work highlights critical security gaps in enterprise RAG deployments and offers practical guidelines to mitigate data leakage, integrity violations, and operational disruption in real-world settings.

Abstract

Retrieval augmented generation (RAG) is a process where a large language model (LLM) retrieves useful information from a database and then generates the responses. It is becoming popular in enterprise settings for daily business operations. For example, Copilot for Microsoft 365 has accumulated millions of businesses. However, the security implications of adopting such RAG-based systems are unclear. In this paper, we introduce ConfusedPilot, a class of security vulnerabilities of RAG systems that confuse Copilot and cause integrity and confidentiality violations in its responses. First, we investigate a vulnerability that embeds malicious text in the modified prompt in RAG, corrupting the responses generated by the LLM. Second, we demonstrate a vulnerability that leaks secret data, which leverages the caching mechanism during retrieval. Third, we investigate how both vulnerabilities can be exploited to propagate misinformation within the enterprise and ultimately impact its operations, such as sales and manufacturing. We also discuss the root cause of these attacks by investigating the architecture of a RAG-based system. This study highlights the security vulnerabilities in today's RAG-based systems and proposes design guidelines to secure future RAG-based systems.

Figures (7)

• Figure 1: High-level architecture of a RAG.
• Figure 2: Retrieval mechanism of a RAG.
• Figure 3: Overview of attacks on Copilot's retrieval mechanism.
• Figure 4: RAG Designs susceptible to phantom resources.
• Figure 5: Bob uses the malicious document to generate and share his documents with others within the enterprise.