Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systems

Robert Friel, Masha Belyi, Atindriyo Sanyal

TL;DR

RAGBench provides the first large-scale, cross-domain benchmark for Retrieval-Augmented Generation evaluation, pairing 100k examples with a formal TRACe metric suite (Utilization, Relevance, Adherence, Completeness). It combines diverse component datasets, standardized RAG formatting, and LLM-based annotations with strong alignment to human judgments. The study demonstrates that a 400M-parameter DeBERTa model finetuned on RAGBench can rival billion-parameter LLM judges on several domains, while also highlighting the persistent difficulty of accurately estimating context relevance. By releasing data, prompts, and post-processing tooling, RAGBench aims to standardize RAG evaluation and guide future improvements in grounding, context usage, and answer completeness across real-world applications.

Abstract

Retrieval-Augmented Generation (RAG) has become a standard architectural pattern for incorporating domain-specific knowledge into user-facing chat applications powered by Large Language Models (LLMs). RAG systems are characterized by (1) a document retriever that queries a domain-specific corpus for context information relevant to an input query, and (2) an LLM that generates a response based on the provided query and context. However, comprehensive evaluation of RAG systems remains a challenge due to the lack of unified evaluation criteria and annotated datasets. In response, we introduce RAGBench: the first comprehensive, large-scale RAG benchmark dataset of 100k examples. It covers five unique industry-specific domains and various RAG task types. RAGBench examples are sourced from industry corpora such as user manuals, making it particularly relevant for industry applications. Further, we formalize the TRACe evaluation framework: a set of explainable and actionable RAG evaluation metrics applicable across all RAG domains. We release the labeled dataset at https://huggingface.co/datasets/rungalileo/ragbench. RAGBench explainable labels facilitate holistic evaluation of RAG systems, enabling actionable feedback for continuous improvement of production applications. Thorough extensive benchmarking, we find that LLM-based RAG evaluation methods struggle to compete with a finetuned RoBERTa model on the RAG evaluation task. We identify areas where existing approaches fall short and propose the adoption of RAGBench with TRACe towards advancing the state of RAG evaluation systems.

RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systems

TL;DR

RAGBench provides the first large-scale, cross-domain benchmark for Retrieval-Augmented Generation evaluation, pairing 100k examples with a formal TRACe metric suite (Utilization, Relevance, Adherence, Completeness). It combines diverse component datasets, standardized RAG formatting, and LLM-based annotations with strong alignment to human judgments. The study demonstrates that a 400M-parameter DeBERTa model finetuned on RAGBench can rival billion-parameter LLM judges on several domains, while also highlighting the persistent difficulty of accurately estimating context relevance. By releasing data, prompts, and post-processing tooling, RAGBench aims to standardize RAG evaluation and guide future improvements in grounding, context usage, and answer completeness across real-world applications.

Abstract

Retrieval-Augmented Generation (RAG) has become a standard architectural pattern for incorporating domain-specific knowledge into user-facing chat applications powered by Large Language Models (LLMs). RAG systems are characterized by (1) a document retriever that queries a domain-specific corpus for context information relevant to an input query, and (2) an LLM that generates a response based on the provided query and context. However, comprehensive evaluation of RAG systems remains a challenge due to the lack of unified evaluation criteria and annotated datasets. In response, we introduce RAGBench: the first comprehensive, large-scale RAG benchmark dataset of 100k examples. It covers five unique industry-specific domains and various RAG task types. RAGBench examples are sourced from industry corpora such as user manuals, making it particularly relevant for industry applications. Further, we formalize the TRACe evaluation framework: a set of explainable and actionable RAG evaluation metrics applicable across all RAG domains. We release the labeled dataset at https://huggingface.co/datasets/rungalileo/ragbench. RAGBench explainable labels facilitate holistic evaluation of RAG systems, enabling actionable feedback for continuous improvement of production applications. Thorough extensive benchmarking, we find that LLM-based RAG evaluation methods struggle to compete with a finetuned RoBERTa model on the RAG evaluation task. We identify areas where existing approaches fall short and propose the adoption of RAGBench with TRACe towards advancing the state of RAG evaluation systems.
Paper Structure (53 sections, 4 equations, 4 figures, 6 tables)

This paper contains 53 sections, 4 equations, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. RAG evaluation
  4. Finetuned RAG evaluation models
  5. RAGBench Construction
  6. Component Datasets
  7. Source Domains
  8. Context Token Length
  9. Task Types
  10. Question Sources
  11. Response Generation
  12. Data Splits
  13. TRACe Evaluation Framework
  14. Definitions
  15. Context Relevance
  16. ...and 38 more sections

Figures (4)

  • Figure 1: RAG system workflow, with highlighted variable parameters: (1) Context format and length, (2) retriever model, (3) number of retrieved documents, and (4) generation model.
  • Figure 2: Example of RAG Question, Context, and Response. Relevant context spans are highlighted, and utilized spans are underlined.
  • Figure 3: Distributions of relevance, utilization, and completeness labels in RAGBench. Y-axis is normalized to visualize densities.
  • Figure 4: Relationship between RAG system configuration and TRACe metrics. (a) The choice and configuration of RAG retriever component affects the average relevance of the retrieved context. In this example, a dense retriever with a low number of documents per query (k=2) yields the highest average context relevance. (b) The choice of LLM and generation prompt affect how well the RAG system utilizes the provided context. Prompting the LLM with a detailed chain-of-thought prompt leads to reduced hallucinations, as well as higher response utilization (c) and completeness (d) rates.