Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Diversity Enhances an LLM's Performance in RAG and Long-context Task

Zhichao Wang, Bin Bi, Yanqi Luo, Sitaram Asur, Claire Na Cheng

TL;DR

The paper addresses the context window bottleneck in LLMs for retrieval-augmented generation and long-context tasks. It proposes diversity-aware content selection using Maximal Marginal Relevance (MMR) and Farthest Point Sampling (FPS), integrated with encoder-based embeddings to select sentences or chunks before LLM processing. Empirical results show that incorporating diversity increases recall of relevant content prior to LLMs, with MMR generally delivering faster latency than FPS and chunk-level, original-order selections yielding strongest downstream performance. These findings highlight the importance of diversity in retrieval for improving Q&A and summarization in long-context settings and guide future development of RAG and long-document systems.

Abstract

The rapid advancements in large language models (LLMs) have highlighted the challenge of context window limitations, primarily due to the quadratic time complexity of the self-attention mechanism (\(O(N^2)\), where \(N\) denotes the context window length). This constraint impacts tasks such as retrieval-augmented generation (RAG) in question answering (Q\&A) and long context summarization. A common approach involves selecting content with the highest similarity to the query; however, this often leads to redundancy and the exclusion of diverse yet relevant information. Building on principles from Maximal Marginal Relevance (MMR) and Farthest Point Sampling (FPS), we integrate diversity into the content selection process. Our findings reveal that incorporating diversity substantially increases the recall of selecting relevant sentences or chunks before LLM-based Q\&A and summarization. These results highlight the importance of maintaining diversity in future LLM applications to further improve summarization and Q\&A outcomes.

Diversity Enhances an LLM's Performance in RAG and Long-context Task

TL;DR

The paper addresses the context window bottleneck in LLMs for retrieval-augmented generation and long-context tasks. It proposes diversity-aware content selection using Maximal Marginal Relevance (MMR) and Farthest Point Sampling (FPS), integrated with encoder-based embeddings to select sentences or chunks before LLM processing. Empirical results show that incorporating diversity increases recall of relevant content prior to LLMs, with MMR generally delivering faster latency than FPS and chunk-level, original-order selections yielding strongest downstream performance. These findings highlight the importance of diversity in retrieval for improving Q&A and summarization in long-context settings and guide future development of RAG and long-document systems.

Abstract

The rapid advancements in large language models (LLMs) have highlighted the challenge of context window limitations, primarily due to the quadratic time complexity of the self-attention mechanism (\(O(N^2)\), where denotes the context window length). This constraint impacts tasks such as retrieval-augmented generation (RAG) in question answering (Q\&A) and long context summarization. A common approach involves selecting content with the highest similarity to the query; however, this often leads to redundancy and the exclusion of diverse yet relevant information. Building on principles from Maximal Marginal Relevance (MMR) and Farthest Point Sampling (FPS), we integrate diversity into the content selection process. Our findings reveal that incorporating diversity substantially increases the recall of selecting relevant sentences or chunks before LLM-based Q\&A and summarization. These results highlight the importance of maintaining diversity in future LLM applications to further improve summarization and Q\&A outcomes.

Paper Structure

This paper contains 21 sections, 2 equations, 6 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. FPS and MMR for Diversity
  4. FPS
  5. MMR
  6. Combine MMR and Modified FPS with LLM for Diversity on Q&A and Summarization
  7. Q&A
  8. Summarization
  9. Experiments
  10. Single Document Q&A
  11. Mutiple Documents Q&A
  12. Sentence and Chunk Splitter Comparison on SquAD
  13. Single Document Summarization
  14. Related Works
  15. Surpass the Context Window Limitation
  16. ...and 6 more sections

Figures (6)

  • Figure 1: For both Q&A and summarization tasks, the initial dataset is divided into sentences or chunks, and corresponding embeddings are extracted. In a traditional pipeline, query embeddings are generated and used to select relevant materials to LLMs for downstream tasks. In contrast, methods like MMR and FPS incorporate diversity in a greedy manner when selecting relevant sentences. This approach increases the recall of the correct answer within the chosen sentences or chunks.
  • Figure 2: An example of FPS: selecting 50 points from 1000 points in 2D space. red: currently selected point; green: previous selected points; blue: unselected points
  • Figure 3: The impact of different hyperparameters: $\alpha$, $w$, $c_{r}$ on the recall accuracy and latency using FPS of the Natural Question dataset of single document Q&A.
  • Figure 4: Prompts for Q&A
  • Figure 5: Prompts for Summarization
  • ...and 1 more figures