Learning to Erase Private Knowledge from Multi-Documents for Retrieval-Augmented Large Language Models

TL;DR

This work tackles privacy leakage in retrieval-augmented generation by introducing Eraser4RAG, a system that rewrites retrieved documents to erase user-defined private knowledge while preserving essential public knowledge. It combines a global knowledge-graph framework with supervised fine-tuning and PPO-based reinforcement learning to balance de-anonymization resistance with downstream task utility. The approach demonstrates superior private-content removal and robust public-content retention across four QA datasets, with favorable transferability to new domains. The solution advances practical privacy protections for RAG while maintaining retrieval effectiveness, offering a scalable path toward privacy-aware information retrieval and generation.

Abstract

Retrieval-Augmented Generation (RAG) is a promising technique for applying LLMs to proprietary domains. However, retrieved documents may contain sensitive knowledge, posing risks of privacy leakage in generative results. Thus, effectively erasing private information from retrieved documents is a key challenge for RAG. Unlike traditional text anonymization, RAG should consider: (1) the inherent multi-document reasoning may face de-anonymization attacks; (2) private knowledge varies by scenarios, so users should be allowed to customize which information to erase; (3) preserving sufficient publicly available knowledge for generation tasks. This paper introduces the privacy erasure task for RAG and proposes Eraser4RAG, a private knowledge eraser which effectively removes user-defined private knowledge from documents while preserving sufficient public knowledge for generation. Specifically, we first construct a global knowledge graph to identify potential knowledge across documents, aiming to defend against de-anonymization attacks. Then we randomly split it into private and public sub-graphs, and fine-tune Flan-T5 to rewrite the retrieved documents excluding private triples. Finally, PPO algorithm optimizes the rewriting model to minimize private triples and maximize public triples retention. Experiments on four QA datasets demonstrate that Eraser4RAG achieves superior erase performance than GPT-4o.

Figures (5)

• Figure 1: Example of de-anonymization through multiple documents reasoning.
• Figure 2: RAG pipeline with privacy erasure.
• Figure 3: The training process of Eraser4RAG.
• Figure 4: The annotation prompt of GPT-4o.
• Figure 5: Ablation study results on different datasets for metrics: (a) $r_{\text{pri}}$, (b) $r_{\text{pub}}$, and (c) $r_{\text{connect}}$. The comparisons include variants without RL (w/o RL), without public triplets (w/o pub), without reward related to public information (w/o pub reward), only inputting triplets from the individual document (each), and our full method (our).