Differentially Private Synthetic Text Generation for Retrieval-Augmented Generation (RAG)

TL;DR

DP-SynRAG tackles privacy challenges in retrieval-augmented generation by generating differentially private synthetic RAG databases before querying. It combines a two-stage DP pipeline: private, locality-preserving clustering of documents and DP synthetic text generation via private prediction, followed by self-filtering to boost downstream RAG utility. The approach yields a fixed privacy budget regardless of the number of queries and outperforms state-of-the-art private RAG baselines while preserving essential topic- and entity-level information. This framework offers a scalable, privacy-preserving alternative for RAG in sensitive domains, with practical implications for medical, recommender, and QA systems.

Abstract

Retrieval-Augmented Generation (RAG) enhances large language models (LLMs) by grounding them in external knowledge. However, its application in sensitive domains is limited by privacy risks. Existing private RAG methods typically rely on query-time differential privacy (DP), which requires repeated noise injection and leads to accumulated privacy loss. To address this issue, we propose DP-SynRAG, a framework that uses LLMs to generate differentially private synthetic RAG databases. Unlike prior methods, the synthetic text can be reused once created, thereby avoiding repeated noise injection and additional privacy costs. To preserve essential information for downstream RAG tasks, DP-SynRAG extends private prediction, which instructs LLMs to generate text that mimics subsampled database records in a DP manner. Experiments show that DP-SynRAG achieves superior performanec to the state-of-the-art private RAG systems while maintaining a fixed privacy budget, offering a scalable solution for privacy-preserving RAG.

Figures (3)

• Figure 1: A demonstration of privacy risks in RAG databases: sensitive information contained in retrieved documents (e.g., patient names) may be revealed to benign users through LLM’s responses.
• Figure 2: A two stage pipeline of DP-SynRAG. Stage 1 first constructs a noisy histogram from the $K$ keywords extracted from each document (a). Each document is assigned to up to $L$ clusters formed by the top-$R$ keywords from the histogram (b). From these clusters, relevant subsets are retrieved using embeddings (c). Stage 2 generates DP synthetic text by rephrasing the documents in each subset and privately aggregating the clipped output token logits (d). Finally, the LLM filters the synthetic texts based on their usefulness for the downstream task (e).
• Figure 3: Accuracy versus number of queries under various fixed total privacy budgets. Since DP-SynRAG can reuse generated synthetic data as a RAG database without incurring additional privacy costs, its accuracy remains constant regardless of the number of queries. In contrast, DP-RAG needs to allocate a smaller privacy budget per query as the number of queries increases, causing its accuracy to decrease significantly.

Theorems & Definitions (11)

• Definition 1: $(\varepsilon,\delta)$-DP
• Theorem 1
• proof : Proof Overview.
• Definition 2: $\rho$-zCDP
• Lemma 2: bun16
• Lemma 3: cesar21
• Lemma 4: Relationship between DP and zCDP bun16
• Lemma 5: Sequential Composition bun16
• Lemma 6: Overlapping Parallel Composition
• proof