Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

EnronQA: Towards Personalized RAG over Private Documents

Michael J. Ryan, Danmei Xu, Chris Nivera, Daniel Campos

TL;DR

EnronQA introduces a large-scale, private-email QA benchmark designed to benchmark retrieval-augmented generation (RAG) in privacy-preserving, personalized settings. The authors present a rigorous, multi-stage dataset construction pipeline (corpus filtering, QA generation, post-processing) to produce 103,638 emails and 528,304 QA pairs across 150 inboxes, enabling exploration of retrieval quality, personalization, and memorization tradeoffs. Through calibration studies and extensive RAG benchmarking, the work shows context augmentation yields linear gains on EnronQA and demonstrates that BM25 can be a strong baseline while larger models improve performance; a case study on LoRA memorization suggests memorization can approach long-context performance but does not yet surpass retrieval-based methods. The resource is released to support continued pretraining, memorization research, and privacy-preserving enterprise LLM deployments, highlighting EnronQA as a foundation for advancing private and personalized information retrieval.

Abstract

Retrieval Augmented Generation (RAG) has become one of the most popular methods for bringing knowledge-intensive context to large language models (LLM) because of its ability to bring local context at inference time without the cost or data leakage risks associated with fine-tuning. A clear separation of private information from the LLM training has made RAG the basis for many enterprise LLM workloads as it allows the company to augment LLM's understanding using customers' private documents. Despite its popularity for private documents in enterprise deployments, current RAG benchmarks for validating and optimizing RAG pipelines draw their corpora from public data such as Wikipedia or generic web pages and offer little to no personal context. Seeking to empower more personal and private RAG we release the EnronQA benchmark, a dataset of 103,638 emails with 528,304 question-answer pairs across 150 different user inboxes. EnronQA enables better benchmarking of RAG pipelines over private data and allows for experimentation on the introduction of personalized retrieval settings over realistic data. Finally, we use EnronQA to explore the tradeoff in memorization and retrieval when reasoning over private documents.

EnronQA: Towards Personalized RAG over Private Documents

TL;DR

EnronQA introduces a large-scale, private-email QA benchmark designed to benchmark retrieval-augmented generation (RAG) in privacy-preserving, personalized settings. The authors present a rigorous, multi-stage dataset construction pipeline (corpus filtering, QA generation, post-processing) to produce 103,638 emails and 528,304 QA pairs across 150 inboxes, enabling exploration of retrieval quality, personalization, and memorization tradeoffs. Through calibration studies and extensive RAG benchmarking, the work shows context augmentation yields linear gains on EnronQA and demonstrates that BM25 can be a strong baseline while larger models improve performance; a case study on LoRA memorization suggests memorization can approach long-context performance but does not yet surpass retrieval-based methods. The resource is released to support continued pretraining, memorization research, and privacy-preserving enterprise LLM deployments, highlighting EnronQA as a foundation for advancing private and personalized information retrieval.

Abstract

Retrieval Augmented Generation (RAG) has become one of the most popular methods for bringing knowledge-intensive context to large language models (LLM) because of its ability to bring local context at inference time without the cost or data leakage risks associated with fine-tuning. A clear separation of private information from the LLM training has made RAG the basis for many enterprise LLM workloads as it allows the company to augment LLM's understanding using customers' private documents. Despite its popularity for private documents in enterprise deployments, current RAG benchmarks for validating and optimizing RAG pipelines draw their corpora from public data such as Wikipedia or generic web pages and offer little to no personal context. Seeking to empower more personal and private RAG we release the EnronQA benchmark, a dataset of 103,638 emails with 528,304 question-answer pairs across 150 different user inboxes. EnronQA enables better benchmarking of RAG pipelines over private data and allows for experimentation on the introduction of personalized retrieval settings over realistic data. Finally, we use EnronQA to explore the tradeoff in memorization and retrieval when reasoning over private documents.
Paper Structure (42 sections, 3 figures, 6 tables)

This paper contains 42 sections, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Retrieval Augmented Generation Benchmarking
  4. Factual Memorization
  5. EnronQA Dataset Construction
  6. Corpus Filtering
  7. Deduplication
  8. Gopher Quality Filters
  9. Language Identification
  10. NSFW/Toxicity Filters
  11. ConcurrentQA
  12. QA Generation Pipeline
  13. Initial Generation
  14. Evaluation
  15. Feedback Generation
  16. ...and 27 more sections

Figures (3)

  • Figure 1: Our multistage compound LLM system for QA Generation on the Enron emails corpus. Our pipeline consists of 4 stages labeled in the diagram above: (1) Initial generation, (2) Evaluation, (3) Feedback generation, and (4) Refinement. Producing a single high-quality question takes 10-50 distinct LLM calls, and the system is optimized end-to-end. Our pipeline asserts that questions are specific, objective, grounded, and high-quality (correlating with human judgment). All Llama icons correspond to Llama3.1 70b Instruct dubey2024llama3herdmodels, the Mistral icon represents the Mixtral-7B-Instruct model jiang2024mixtralexperts, and our retriever is a bi-encoder using Snowflake's artic-embed-m-v1.5 merrick2024embeddingclusteringdataimprove.
  • Figure 2: Question Rewrite Pipeline. First, we ask Llama 3.1 70B Instruct to rewrite the question, and then we ask it to answer this new question. Finally, we use Llama 3.1 70B Instruct to check that the answers match.
  • Figure 3: Calibration experiment results. Although all benchmarks scale roughly linearly with more accurate context, EnronQA is the only benchmark where adding context always outperforms the no-context baseline. For TriviaQA, it takes Recall@1 of nearly 0.85 to surpass the performance of the no-context baseline.