Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Know Your RAG: Dataset Taxonomy and Generation Strategies for Evaluating RAG Systems

Rafael Teixeira de Lima, Shubham Gupta, Cesar Berrospi, Lokesh Mishra, Michele Dolfi, Peter Staar, Panagiotis Vagenas

TL;DR

The paper tackles the challenge of evaluating Retrieval Augmented Generation (RAG) systems by decoupling evaluation data design from downstream tasks. It introduces a label taxonomy for context–query pairs, analyzes public QA datasets to reveal label imbalances, and demonstrates that retrieval performance depends heavily on label composition. To address this, it proposes two data-generation approaches: a grounded statement-extraction method and a cost-efficient fine-tuning pipeline for small LLMs, enabling diverse, labeled Q&A datasets without excessive LLM querying. These contributions equip RAG developers with robust, know-your-data evaluation pipelines, improving the reliability and applicability of local performance assessments across use cases.

Abstract

Retrieval Augmented Generation (RAG) systems are a widespread application of Large Language Models (LLMs) in the industry. While many tools exist empowering developers to build their own systems, measuring their performance locally, with datasets reflective of the system's use cases, is a technological challenge. Solutions to this problem range from non-specific and cheap (most public datasets) to specific and costly (generating data from local documents). In this paper, we show that using public question and answer (Q&A) datasets to assess retrieval performance can lead to non-optimal systems design, and that common tools for RAG dataset generation can lead to unbalanced data. We propose solutions to these issues based on the characterization of RAG datasets through labels and through label-targeted data generation. Finally, we show that fine-tuned small LLMs can efficiently generate Q&A datasets. We believe that these observations are invaluable to the know-your-data step of RAG systems development.

Know Your RAG: Dataset Taxonomy and Generation Strategies for Evaluating RAG Systems

TL;DR

The paper tackles the challenge of evaluating Retrieval Augmented Generation (RAG) systems by decoupling evaluation data design from downstream tasks. It introduces a label taxonomy for context–query pairs, analyzes public QA datasets to reveal label imbalances, and demonstrates that retrieval performance depends heavily on label composition. To address this, it proposes two data-generation approaches: a grounded statement-extraction method and a cost-efficient fine-tuning pipeline for small LLMs, enabling diverse, labeled Q&A datasets without excessive LLM querying. These contributions equip RAG developers with robust, know-your-data evaluation pipelines, improving the reliability and applicability of local performance assessments across use cases.

Abstract

Retrieval Augmented Generation (RAG) systems are a widespread application of Large Language Models (LLMs) in the industry. While many tools exist empowering developers to build their own systems, measuring their performance locally, with datasets reflective of the system's use cases, is a technological challenge. Solutions to this problem range from non-specific and cheap (most public datasets) to specific and costly (generating data from local documents). In this paper, we show that using public question and answer (Q&A) datasets to assess retrieval performance can lead to non-optimal systems design, and that common tools for RAG dataset generation can lead to unbalanced data. We propose solutions to these issues based on the characterization of RAG datasets through labels and through label-targeted data generation. Finally, we show that fine-tuned small LLMs can efficiently generate Q&A datasets. We believe that these observations are invaluable to the know-your-data step of RAG systems development.

Paper Structure

This paper contains 31 sections, 5 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related work:
  3. Label taxonomy
  4. Public Datasets
  5. Labelling examples using LLMs
  6. Retrieval performance across classes
  7. Generating Balanced Datasets
  8. Model Fine Tuning
  9. Synthetic Datasets Quality Comparisons
  10. Conclusion
  11. Public Datasets
  12. SQuAD2:
  13. NewsQA:
  14. PubMedQA:
  15. HotpotQA:
  16. ...and 16 more sections

Figures (5)

  • Figure 1: Composition of labels for different datasets.
  • Figure 2: Distribution of LLama3 labels for statement extraction (top) and fine-tuned model (bottom) per requested label.
  • Figure 3: Average critique ratings per question label for different Q&A generation strategies, for all datasets.
  • Figure 4: Confusion matrix between labels given by Llama-3-70b and the annotators' majority vote. Each row shows the distribution of Llama-3-70b labels given a majority vote label.
  • Figure 5: Distribution of Llama-3-70b labels for questions generated by Ragas using simple and reasoning evolutions