Privacy-Preserving Retrieval-Augmented Generation with Differential Privacy

TL;DR

This work addresses privacy leakage in retrieval-augmented generation by introducing differential-privacy guarantees for RAG outputs. It proposes two algorithms, DPVoteRAG and DPSparseVoteRAG, that partition the external corpus and employ a voting-based token-generation scheme, with DPSparseVoteRAG leveraging the sparse vector technique to spend privacy budget only when sensitive information is necessary. Empirical results across QA datasets and multiple LLMs show that DPSparseVoteRAG improves QA accuracy under a practical DP budget (around $ε ≈ 10$) and outperforms DPVoteRAG, while providing resilience to empirical privacy attacks. The methods offer a practical path to deploying RAG systems on sensitive data, balancing privacy with utility, and lay groundwork for broader DP applications in generation tasks.

Abstract

With the recent remarkable advancement of large language models (LLMs), there has been a growing interest in utilizing them in the domains with highly sensitive data that lies outside their training data. For this purpose, retrieval-augmented generation (RAG) is particularly effective -- it assists LLMs by directly providing relevant information from the external knowledge sources. However, without extra privacy safeguards, RAG outputs risk leaking sensitive information from the external data source. In this work, we explore RAG under differential privacy (DP), a formal guarantee of data privacy. The main challenge with differentially private RAG is how to generate long accurate answers within a moderate privacy budget. We address this by proposing an algorithm that smartly spends privacy budget only for the tokens that require the sensitive information and uses the non-private LLM for other tokens. Our extensive empirical evaluations reveal that our algorithm outperforms the non-RAG baseline under a reasonable privacy budget of $ε\approx 10$ across different models and datasets.

Figures (9)

• Figure 1: Overview of our problem setting. Note that the LLM in RAG is trained outside the privacy barrier by DP.
• Figure 2: Overview of DPSparseVoteRAG. It first retrieves the relevant documents to a question $x$ from sensitive external knowledge $D$ and partitions them uniform randomly (left). When generating $t$-th token (right), it takes a question $x$ and an interim answer up to $t-1$-th tokens, $y_{<t}$, as inputs and outputs a new token $y_t$. Components enclosed in red indicate that operations involve randomness for the privacy guarantee. DPVoteRAG works the same until computing the output token histogram $h_t$, but it immediately finds the majority token in $h_t$ afterwards.
• Figure 3: Average match accuracy comparison across algorithms on Trivia (upper row) and NQ (lower row) datasets with different generator LLMs: OPT (1.3B) (left column), Pythia (1.4B) (middle column), and Llama 3.1 (8B) (right column). The reported results are the means and standard deviations of average match accuracy over three runs. We report the best results over hyperparameters for each $\epsilon_\mathrm{total}$.
• Figure 4: TPR-FPR curve of S$^2$MIA for VoteRAG and DPSparseVoteRAG ($\varepsilon$=10) when the base LLMs are OPT (1.3B), Pythia (1.4B) and Llama 3.1 (8B).
• Figure 5: Average numbers of generated tokens for each $\epsilon_\mathrm{token}$ and $\epsilon_\mathrm{total}$ with OPT (1.3B) on Trivia dataset. We fix $m=50$. We report the means and standard deviations over three runs. We see DPSparseVoteRAG generates much more tokens than DPVoteRAG.

Theorems & Definitions (6)

• Definition 1: $(\epsilon,\delta)$-Differential Privacy dwork_our_2006-1
• Theorem 1
• Theorem 2
• proof
• Theorem 3
• proof