Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Revisiting RAG Retrievers: An Information Theoretic Benchmark

Wenqing Zheng, Dmitri Kalaev, Noah Fatsi, Daniel Barcklow, Owen Reinert, Igor Melnyk, Senthil Kumar, C. Bayan Bruss

TL;DR

It is shown that if chosen carefully, an ensemble of retrievers outperforms any single retriever and a fresh perspective on the structure of modern retrieval techniques and actionable guidance for designing robust and efficient RAG systems is provided.

Abstract

Retrieval-Augmented Generation (RAG) systems rely critically on the retriever module to surface relevant context for large language models. Although numerous retrievers have recently been proposed, each built on different ranking principles such as lexical matching, dense embeddings, or graph citations, there remains a lack of systematic understanding of how these mechanisms differ and overlap. Existing benchmarks primarily compare entire RAG pipelines or introduce new datasets, providing little guidance on selecting or combining retrievers themselves. Those that do compare retrievers directly use a limited set of evaluation tools which fail to capture complementary and overlapping strengths. This work presents MIGRASCOPE, a Mutual Information based RAG Retriever Analysis Scope. We revisit state-of-the-art retrievers and introduce principled metrics grounded in information and statistical estimation theory to quantify retrieval quality, redundancy, synergy, and marginal contribution. We further show that if chosen carefully, an ensemble of retrievers outperforms any single retriever. We leverage the developed tools over major RAG corpora to provide unique insights on contribution levels of the state-of-the-art retrievers. Our findings provide a fresh perspective on the structure of modern retrieval techniques and actionable guidance for designing robust and efficient RAG systems.

Revisiting RAG Retrievers: An Information Theoretic Benchmark

TL;DR

It is shown that if chosen carefully, an ensemble of retrievers outperforms any single retriever and a fresh perspective on the structure of modern retrieval techniques and actionable guidance for designing robust and efficient RAG systems is provided.

Abstract

Retrieval-Augmented Generation (RAG) systems rely critically on the retriever module to surface relevant context for large language models. Although numerous retrievers have recently been proposed, each built on different ranking principles such as lexical matching, dense embeddings, or graph citations, there remains a lack of systematic understanding of how these mechanisms differ and overlap. Existing benchmarks primarily compare entire RAG pipelines or introduce new datasets, providing little guidance on selecting or combining retrievers themselves. Those that do compare retrievers directly use a limited set of evaluation tools which fail to capture complementary and overlapping strengths. This work presents MIGRASCOPE, a Mutual Information based RAG Retriever Analysis Scope. We revisit state-of-the-art retrievers and introduce principled metrics grounded in information and statistical estimation theory to quantify retrieval quality, redundancy, synergy, and marginal contribution. We further show that if chosen carefully, an ensemble of retrievers outperforms any single retriever. We leverage the developed tools over major RAG corpora to provide unique insights on contribution levels of the state-of-the-art retrievers. Our findings provide a fresh perspective on the structure of modern retrieval techniques and actionable guidance for designing robust and efficient RAG systems.
Paper Structure (62 sections, 44 equations, 7 figures, 1 table)

This paper contains 62 sections, 44 equations, 7 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. RAG Benchmarking
  4. RAG Ensemble Methods
  5. Information-Theoretic Analysis in RAG
  6. Analytical Framework
  7. Preliminaries: GraphRAG Formulations
  8. Information-Theoretic View of RAG
  9. Mutual information (MI).
  10. Divergence.
  11. Pointwise mutual information in RAG.
  12. Pseudo Ground-Truth Chunk Probability
  13. Golden chunk reinforcement.
  14. Retriever Divergence Score
  15. Ensembling Multiple Retrievers
  16. ...and 47 more sections

Figures (7)

  • Figure 1: Metric Correlations Across Datasets, Legend by Retriever
  • Figure 2: Hparam robustness visualization. Using the first bar of the first subfigure (RAG) to interpret the meaning: in HotPotQA dataset and for GRAG-Window retriever, the Pearson correlation between the Div metric and the Recall is 0.54 if the Div is computed at top 3 chunks retrieved by all retrievers (otherwise, top 3 chunks retrieved only by HippoRAG) and $\gamma$ is set to 2 for the Div.
  • Figure 4: Retriever redundancy/synergy spectrum.
  • Figure 3.1: Shapley share for the best ensemble configs.
  • Figure 3.2: Recall under ensemble perturbation.
  • ...and 2 more figures