Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach

Zhuowan Li, Cheng Li, Mingyang Zhang, Qiaozhu Mei, Michael Bendersky

TL;DR

The paper rigorously compares Retrieval Augmented Generation (RAG) and long-context LLMs across diverse public datasets and modern models, revealing that long-context models typically outperform RAG when adequately resourced, while RAG maintains a clear cost advantage. It introduces Self-Route, a simple two-step routing approach driven by model self-reflection that routes queries to RAG or LC to achieve LC-like performance at a fraction of the cost. The analysis covers dataset and model setups, retriever choices, failures, and synthetic data biases, providing practical guidance and revealing substantial token-cost savings (e.g., up to 65% for some OpenAI models). Overall, the work demonstrates the value of hybrid, self-aware routing for scalable long-context reasoning in real-world applications.

Abstract

Retrieval Augmented Generation (RAG) has been a powerful tool for Large Language Models (LLMs) to efficiently process overly lengthy contexts. However, recent LLMs like Gemini-1.5 and GPT-4 show exceptional capabilities to understand long contexts directly. We conduct a comprehensive comparison between RAG and long-context (LC) LLMs, aiming to leverage the strengths of both. We benchmark RAG and LC across various public datasets using three latest LLMs. Results reveal that when resourced sufficiently, LC consistently outperforms RAG in terms of average performance. However, RAG's significantly lower cost remains a distinct advantage. Based on this observation, we propose Self-Route, a simple yet effective method that routes queries to RAG or LC based on model self-reflection. Self-Route significantly reduces the computation cost while maintaining a comparable performance to LC. Our findings provide a guideline for long-context applications of LLMs using RAG and LC.

Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach

TL;DR

The paper rigorously compares Retrieval Augmented Generation (RAG) and long-context LLMs across diverse public datasets and modern models, revealing that long-context models typically outperform RAG when adequately resourced, while RAG maintains a clear cost advantage. It introduces Self-Route, a simple two-step routing approach driven by model self-reflection that routes queries to RAG or LC to achieve LC-like performance at a fraction of the cost. The analysis covers dataset and model setups, retriever choices, failures, and synthetic data biases, providing practical guidance and revealing substantial token-cost savings (e.g., up to 65% for some OpenAI models). Overall, the work demonstrates the value of hybrid, self-aware routing for scalable long-context reasoning in real-world applications.

Abstract

Retrieval Augmented Generation (RAG) has been a powerful tool for Large Language Models (LLMs) to efficiently process overly lengthy contexts. However, recent LLMs like Gemini-1.5 and GPT-4 show exceptional capabilities to understand long contexts directly. We conduct a comprehensive comparison between RAG and long-context (LC) LLMs, aiming to leverage the strengths of both. We benchmark RAG and LC across various public datasets using three latest LLMs. Results reveal that when resourced sufficiently, LC consistently outperforms RAG in terms of average performance. However, RAG's significantly lower cost remains a distinct advantage. Based on this observation, we propose Self-Route, a simple yet effective method that routes queries to RAG or LC based on model self-reflection. Self-Route significantly reduces the computation cost while maintaining a comparable performance to LC. Our findings provide a guideline for long-context applications of LLMs using RAG and LC.
Paper Structure (20 sections, 4 figures, 9 tables)

This paper contains 20 sections, 4 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Benchmarking RAG versus LC
  4. Datasets and metrics
  5. Models and Retrievers
  6. Benchmarking results
  7. Self-Route
  8. Motivation
  9. Self-Route
  10. Results
  11. Analysis
  12. Ablations of k
  13. Why does RAG fail?
  14. Different retrievers
  15. Results on synthetic data
  16. ...and 5 more sections

Figures (4)

  • Figure 1: While long-context LLMs (LC) surpass RAG in long-context understanding, RAG is significantly more cost-efficient. Our approach, Self-Route, combining RAG and LC, achieves comparable performance to LC at a much lower cost.
  • Figure 2: Distribution of the difference of prediction scores between RAG and LC (computed w.r.t. groundtruth labels). RAG and LC predictions are highly identical, for both correct and incorrect ones.
  • Figure 3: Trade-off curves between (a) model performance and (b) token percentage as a function of $k$.
  • Figure 4: Distribution of typical RAG failure reasons.