Differentially Private Retrieval-Augmented Generation

TL;DR

DP-KSA is presented, a novel privacy-preserving RAG algorithm that integrates DP using the propose-test-release paradigm and formally shows that DP-KSA provides formal DP guarantees on the generated output with respect to the RAG database.

Abstract

Retrieval-augmented generation (RAG) is a widely used framework for reducing hallucinations in large language models (LLMs) on domain-specific tasks by retrieving relevant documents from a database to support accurate responses. However, when the database contains sensitive corpora, such as medical records or legal documents, RAG poses serious privacy risks by potentially exposing private information through its outputs. Prior work has demonstrated that one can practically craft adversarial prompts that force an LLM to regurgitate the augmented contexts. A promising direction is to integrate differential privacy (DP), a privacy notion that offers strong formal guarantees, into RAG systems. However, naively applying DP mechanisms into existing systems often leads to significant utility degradation. Particularly for RAG systems, DP can reduce the usefulness of the augmented contexts leading to increase risk of hallucination from the LLMs. Motivated by these challenges, we present DP-KSA, a novel privacy-preserving RAG algorithm that integrates DP using the propose-test-release paradigm. DP-KSA follows from a key observation that most question-answering (QA) queries can be sufficiently answered with a few keywords. Hence, DP-KSA first obtains an ensemble of relevant contexts, each of which will be used to generate a response from an LLM. We utilize these responses to obtain the most frequent keywords in a differentially private manner. Lastly, the keywords are augmented into the prompt for the final output. This approach effectively compresses the semantic space while preserving both utility and privacy. We formally show that DP-KSA provides formal DP guarantees on the generated output with respect to the RAG database. We evaluate DP-KSA on two QA benchmarks using three instruction-tuned LLMs, and our empirical results demonstrate that DP-KSA achieves a strong privacy-utility tradeoff.

Figures (9)

• Figure 1: Overview of the DP RAG problem setting. The adversarial bound illustrates what capabilities the adversary has. In this case, the adversary can only query the RAG system and access the answer. The generator model has been pre-trained on publicly available data, which the adversary has access to. However, we are not concerned about preserving the privacy of the pre-training data.
• Figure 2: The proposed DP-KSA framework consists of three steps. First, it retrieves the top-$N$ documents most relevant to the query $\mathbf{x}$ from the private database $D$. Each retrieved document $D^{\mathbf{x}}_i$ is paired with query $\mathbf{x}$ and passed to the generator model $F$ to produce responses. Next, DP-KSA applies a differentially private mechanism to extract the most frequent keywords from the ensemble of responses. Finally, the selected keywords are combined with query $\mathbf{x}$ and fed back into the generator $F$ to produce the final output $\mathbf{y}$.
• Figure 3: Histogram of token lengths of ground truth answers in NQ dataset.
• Figure 4: Histogram of token lengths of ground truth answers in TQA dataset.
• Figure 5: Results of DP-KSA on NQ dataset with different generator LLMs: Qwen 2.5 (3B), Llama 3.2 (3B), and Llama 3.1 (8B). We use three baselines including non-RAG ($\epsilon=0$), non-private RAG with top-2 retrieved documents ($\epsilon=\infty$), and non-private KSA ($\epsilon=\infty$).

Theorems & Definitions (15)

• Definition 2.1: $(\epsilon, \delta$)-DP dp
• Theorem 5.1
• Remark 5.2
• Definition A.1: Rényi Divergence
• Definition A.2
• Theorem A.3: Composition mironov2017renyi
• Theorem A.4: Post-Processing mironov2017renyi
• Definition A.5: Approximate RDP bun2016concentratedzhu2022adaptive
• Theorem A.6: Conversion from approximate RDP to Approximate DP zhu2022adaptive
• Definition A.7: Exponential Mechanism mcsherry2007mechanism