Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Is Agentic RAG worth it? An experimental comparison of RAG approaches

Pietro Ferrazzi, Milica Cvjeticanin, Alessio Piraccini, Davide Giannuzzi

TL;DR

This work empirically compares Enhanced RAG and Agentic RAG across four evaluation dimensions—user intent handling, query-documents alignment, retrieved document refinement, and underlying LLM quality—using four datasets spanning QA and IR/E tasks. It finds that Agentic RAG better handles user intent in narrow, well-defined domains, while Enhanced RAG offers more robust retrieval and document alignment in broader, noisier settings; Agentic’s dynamic rewriting can improve retrieval relevance, but its iterative nature incurs higher costs. The results highlight that neither paradigm is universally superior and that model size impacts both similarly, underscoring important cost-performance tradeoffs. The study provides practical guidance for practitioners on selecting and potentially hybridizing RAG designs, emphasizing efficiency considerations and the potential value of incorporating explicit re-ranking within Agentic workflows as a path to gains.

Abstract

Retrieval-Augmented Generation (RAG) systems are usually defined by the combination of a generator and a retrieval component that extracts textual context from a knowledge base to answer user queries. However, such basic implementations exhibit several limitations, including noisy or suboptimal retrieval, misuse of retrieval for out-of-scope queries, weak query-document matching, and variability or cost associated with the generator. These shortcomings have motivated the development of "Enhanced" RAG, where dedicated modules are introduced to address specific weaknesses in the workflow. More recently, the growing self-reflective capabilities of Large Language Models (LLMs) have enabled a new paradigm, which we refer to as "Agentic" RAG. In this approach, the LLM orchestrates the entire process-deciding which actions to perform, when to perform them, and whether to iterate-thereby reducing reliance on fixed, manually engineered modules. Despite the rapid adoption of both paradigms, it remains unclear which approach is preferable under which conditions. In this work, we conduct an extensive, empirically driven evaluation of Enhanced and Agentic RAG across multiple scenarios and dimensions. Our results provide practical insights into the trade-offs between the two paradigms, offering guidance on selecting the most effective RAG design for real-world applications, considering both costs and performance.

Is Agentic RAG worth it? An experimental comparison of RAG approaches

TL;DR

This work empirically compares Enhanced RAG and Agentic RAG across four evaluation dimensions—user intent handling, query-documents alignment, retrieved document refinement, and underlying LLM quality—using four datasets spanning QA and IR/E tasks. It finds that Agentic RAG better handles user intent in narrow, well-defined domains, while Enhanced RAG offers more robust retrieval and document alignment in broader, noisier settings; Agentic’s dynamic rewriting can improve retrieval relevance, but its iterative nature incurs higher costs. The results highlight that neither paradigm is universally superior and that model size impacts both similarly, underscoring important cost-performance tradeoffs. The study provides practical guidance for practitioners on selecting and potentially hybridizing RAG designs, emphasizing efficiency considerations and the potential value of incorporating explicit re-ranking within Agentic workflows as a path to gains.

Abstract

Retrieval-Augmented Generation (RAG) systems are usually defined by the combination of a generator and a retrieval component that extracts textual context from a knowledge base to answer user queries. However, such basic implementations exhibit several limitations, including noisy or suboptimal retrieval, misuse of retrieval for out-of-scope queries, weak query-document matching, and variability or cost associated with the generator. These shortcomings have motivated the development of "Enhanced" RAG, where dedicated modules are introduced to address specific weaknesses in the workflow. More recently, the growing self-reflective capabilities of Large Language Models (LLMs) have enabled a new paradigm, which we refer to as "Agentic" RAG. In this approach, the LLM orchestrates the entire process-deciding which actions to perform, when to perform them, and whether to iterate-thereby reducing reliance on fixed, manually engineered modules. Despite the rapid adoption of both paradigms, it remains unclear which approach is preferable under which conditions. In this work, we conduct an extensive, empirically driven evaluation of Enhanced and Agentic RAG across multiple scenarios and dimensions. Our results provide practical insights into the trade-offs between the two paradigms, offering guidance on selecting the most effective RAG design for real-world applications, considering both costs and performance.
Paper Structure (49 sections, 2 equations, 9 figures, 9 tables)

This paper contains 49 sections, 2 equations, 9 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Retrieval Augmented Generation
  3. RAG definition
  4. Naïve RAG
  5. Enhanced RAG
  6. Agentic RAG
  7. Related work
  8. RAG
  9. Agents design and implementation
  10. The Need for Empirical Comparison
  11. Data
  12. Evaluation
  13. User intent handling
  14. Definition
  15. Enhanced Implementation
  16. ...and 34 more sections

Figures (9)

  • Figure 1: Left — Enhanced RAG. The system is composed by a sequence of modules, each responsible for improving a specific stage of the RAG pipeline. A router determines whether a query should trigger retrieval; a rewriter reformulates the query; a retriever selects candidate chunks from the knowledge base; and a reranker orders the retrieved context before passing it to the generator. The workflow is fixed: information flows through predefined blocks intended to mitigate known weaknesses of naïve RAG systems (defined by the simple composition of the retriever and generator blocks). Right — Agentic RAG. The LLM acts as an agent that orchestrates the entire process. At each step, it can choose whether to call a RAG tool or proceed directly to answer generation. Retrieval and context refinement can be repeated, and the agent autonomously selects and sequences operations based on the evolving state of the task. This yields an iterative pipeline without predefined intermediate modules.
  • Figure 2: Performances of Enhanced and Agentic settings when changing the underlying LLM. For FIQA, the metric is the overall ratio of the classification metric (1 if the answer follows the instruction, 0 otherwise). For CQADupStack-EN, the metric is based on a pairwise analysis, calculated as the ratio of times when the larger model's answer is better than that of the smaller counterpart. Both metrics are calculated via LLM-as-a-Judge (Selene-70B).
  • Figure 3: System prompt template that defines the agent for all the tasks.
  • Figure 4: System prompt filling values that define each of the four selected tasks.
  • Figure 5: Answer node system prompts that define the answer generation.
  • ...and 4 more figures