Privacy-Preserving Reasoning with Knowledge-Distilled Parametric Retrieval Augmented Generation

TL;DR

DistilledPRAG tackles privacy-preserving reasoning by removing plaintext exposure in retrieval-augmented generation and aligning parametric RAG with standard RAG through knowledge distillation. It synthesizes cross-document QA data, masks plaintext documents, and trains a parameter generator to produce LoRA updates that emulate RAG-style reasoning without access to the original content. The method achieves strong in-domain and OOD performance across four QA datasets, with ablations showing the necessity of alignment losses, cross-document data, and a carefully initialized special masking token. Results demonstrate competitive latency and robust generalization, highlighting the practical potential of privacy-preserving parametric retrieval for open-domain QA.

Abstract

The current RAG system requires uploading plaintext documents to the cloud, risking private data leakage. Parametric RAG (PRAG) encodes documents as LoRA parameters within LLMs, offering a possible way to reduce exposure of raw content. However, it still faces two issues: (1) PRAG demands synthesizing QA pairs and fine-tuning LLM for each individual document to create its corresponding LoRA, leading to unacceptable inference latency. (2) The performance of PRAG relies solely on synthetic QA data while lacking internal alignment with standard RAG, resulting in poor generalization on out-of-distribution(OOD) inputs. Therefore, achieving high-efficiency parameterization while maintaining RAG-level performance remains a critical challenge for privacy-preserving reasoning. In this paper, we propose DistilledPRAG, a generalizable knowledge-distilled parametric RAG model aligned with standard RAG in document structure and parameter activation. We first synthesize QA pairs from single and multi-documents to enhance cross-document reasoning. Then, we mask the plaintext documents with a special token and translate them to LoRA via a parameter generator, maintaining the standard RAG document structure. Finally, guided by synthetic QA data, we train the parameter generator to match standard RAG's hidden states and output logits, enabling RAG-style reasoning without original documents. Experiments on four QA datasets show that DistilledPRAG outperforms baselines in accuracy and generalizes well on OOD data.

Figures (9)

• Figure 1: Inference Paradigms for standard RAG, PRAG, DyPRAG, and our DistilledPRAG. (1) Standard RAG inputs the plaintext documents and the question. (2) PRAG generates QA pairs per document to fine-tune LoRA adapters, and sums them to obtain document aggregated representations for LLM injection. (3) DyPRAG translates individual documents to its LoRA and averages them to achieve document aggregation for LLM injection. (4) DistilledPRAG generates cross-document LoRA from concatenated documents via a parameter generator, then takes masked documents and questions as input, similar to standard RAG.
• Figure 2: The Architecture of DistilledPRAG Model. 1Use DeepSeek-V3 to mine knowledge from a single document and augmented cross-documents by random concatenation. 2Train a parameter generator to map documents to a LoRA for student LLM, enabling it to mimic a teacher RAG's reasoning by minimizing differences in hidden states and logits on synthetic data.
• Figure 3: Prompt for single-document QA synthesis.
• Figure 4: Prompt for single-document QA synthesis.
• Figure 5: Prompt for inference.