Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Scaling DPPs for RAG: Density Meets Diversity

Xun Sun, Baiheng Xie, Li Huang, Qiang Gao

Abstract

Retrieval-Augmented Generation (RAG) enhances Large Language Models (LLMs) by grounding generation in external knowledge, yielding relevance responses that are aligned with factual evidence and evolving corpora. Standard RAG pipelines construct context through relevance ranking, performing point-wise scoring between the user query and each corpora chunk. This formulation, however, ignores interactions among retrieved candidates, leading to redundant contexts that dilute density and fail to surface complementary evidence. We argue that effective retrieval should optimize jointly for both density and diversity, ensuring the grounding evidence that is dense in information yet diverse in coverage. In this study, we propose ScalDPP, a diversity-aware retrieval mechanism for RAG that incorporates Determinantal Point Processes (DPPs) through a lightweight P-Adapter, enabling scalable modeling of inter-chunk dependencies and complementary context selection. In addition, we develop a novel set-level objective, Diverse Margin Loss (DML), that enforces ground-truth complementary evidence chains to dominate any equally sized redundant alternatives under DPP geometry. Experimental results demonstrate the superiority of ScalDPP, substantiating our core statement in practice.

Scaling DPPs for RAG: Density Meets Diversity

Abstract

Retrieval-Augmented Generation (RAG) enhances Large Language Models (LLMs) by grounding generation in external knowledge, yielding relevance responses that are aligned with factual evidence and evolving corpora. Standard RAG pipelines construct context through relevance ranking, performing point-wise scoring between the user query and each corpora chunk. This formulation, however, ignores interactions among retrieved candidates, leading to redundant contexts that dilute density and fail to surface complementary evidence. We argue that effective retrieval should optimize jointly for both density and diversity, ensuring the grounding evidence that is dense in information yet diverse in coverage. In this study, we propose ScalDPP, a diversity-aware retrieval mechanism for RAG that incorporates Determinantal Point Processes (DPPs) through a lightweight P-Adapter, enabling scalable modeling of inter-chunk dependencies and complementary context selection. In addition, we develop a novel set-level objective, Diverse Margin Loss (DML), that enforces ground-truth complementary evidence chains to dominate any equally sized redundant alternatives under DPP geometry. Experimental results demonstrate the superiority of ScalDPP, substantiating our core statement in practice.

Paper Structure

This paper contains 16 sections, 35 equations, 5 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Method
  3. DPP-based Subset Selection
  4. P-Adapter
  5. Diverse Margin Loss
  6. Experiments
  7. Related Work
  8. Conclusion
  9. Availability
  10. Derivation of Fast Greedy MAP Inference
  11. Proofs
  12. LSE Approximation of Maximum Function
  13. Proof that the Approximation is a Convex Upper Bound
  14. Step 1: Prove the Upper Bound ($\tilde{\mathcal{L}} \geq \mathcal{L}$)
  15. Step 2: Prove Convexity of $\tilde{\mathcal{L}}$
  16. ...and 1 more sections

Figures (5)

  • Figure 1: Standard RAG models query-chunk relevance but neglects inter-chunk diversity and complementarity. Example from MultiHop-RAG tang2024multihoprag.
  • Figure 2: Overview of the ScalDPP approach. The pipeline integrates dynamic DPP subset selection with adaptive embeddings to achieve complementary chunk selection.
  • Figure 3: Time consumption analysis.
  • Figure 4: Training curves for DML and NLL with/without reranker.
  • Figure 5: Case study on multi-hop queries. Left: t-SNE projections of chunk embeddings for 2-, 3-, and 4-hop cases; top row: Standard RAG, bottom row: ScalDPP. Right: Zoom-in on the 3-hop query showing the query text and the top-3 retrieved chunks. Standard RAG selects only one ground-truth chunk in its top-3, whereas ScalDPP precisely recovers all three required positive evidence chunks.