Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Generating Diverse Q&A Benchmarks for RAG Evaluation with DataMorgana

Simone Filice, Guy Horowitz, David Carmel, Zohar Karnin, Liane Lewin-Eytan, Yoelle Maarek

TL;DR

The paper tackles the challenge of evaluating Retrieval-Augmented Generation (RAG) systems in domain-specific settings by addressing the lack of diverse synthetic benchmarks. It introduces DataMorgana, a two-stage, configurable pipeline that defines multiple end-user and question categories to drive LLM-based Q&A generation via prompts. Through experiments on the CORD-19 and open-NQ corpora, DataMorgana demonstrates superior lexical, syntactic, and semantic diversity compared with baselines, with ablations showing that question categorization is a major diversity driver. The approach enables rapid, scalable generation of realistic and varied benchmarks for robust RAG evaluation and is poised for beta release to select research teams for SIGIR 2025 LiveRAG.

Abstract

Evaluating Retrieval-Augmented Generation (RAG) systems, especially in domain-specific contexts, requires benchmarks that address the distinctive requirements of the applicative scenario. Since real data can be hard to obtain, a common strategy is to use LLM-based methods to generate synthetic data. Existing solutions are general purpose: given a document, they generate a question to build a Q&A pair. However, although the generated questions can be individually good, they are typically not diverse enough to reasonably cover the different ways real end-users can interact with the RAG system. We introduce here DataMorgana, a tool for generating highly customizable and diverse synthetic Q&A benchmarks tailored to RAG applications. DataMorgana enables detailed configurations of user and question categories and provides control over their distribution within the benchmark. It uses a lightweight two-stage process, ensuring efficiency and fast iterations, while generating benchmarks that reflect the expected traffic. We conduct a thorough line of experiments, showing quantitatively and qualitatively that DataMorgana surpasses existing tools and approaches in producing lexically, syntactically, and semantically diverse question sets across domain-specific and general-knowledge corpora. DataMorgana will be made available to selected teams in the research community, as first beta testers, in the context of the upcoming SIGIR'2025 LiveRAG challenge to be announced in early February 2025.

Generating Diverse Q&A Benchmarks for RAG Evaluation with DataMorgana

TL;DR

The paper tackles the challenge of evaluating Retrieval-Augmented Generation (RAG) systems in domain-specific settings by addressing the lack of diverse synthetic benchmarks. It introduces DataMorgana, a two-stage, configurable pipeline that defines multiple end-user and question categories to drive LLM-based Q&A generation via prompts. Through experiments on the CORD-19 and open-NQ corpora, DataMorgana demonstrates superior lexical, syntactic, and semantic diversity compared with baselines, with ablations showing that question categorization is a major diversity driver. The approach enables rapid, scalable generation of realistic and varied benchmarks for robust RAG evaluation and is poised for beta release to select research teams for SIGIR 2025 LiveRAG.

Abstract

Evaluating Retrieval-Augmented Generation (RAG) systems, especially in domain-specific contexts, requires benchmarks that address the distinctive requirements of the applicative scenario. Since real data can be hard to obtain, a common strategy is to use LLM-based methods to generate synthetic data. Existing solutions are general purpose: given a document, they generate a question to build a Q&A pair. However, although the generated questions can be individually good, they are typically not diverse enough to reasonably cover the different ways real end-users can interact with the RAG system. We introduce here DataMorgana, a tool for generating highly customizable and diverse synthetic Q&A benchmarks tailored to RAG applications. DataMorgana enables detailed configurations of user and question categories and provides control over their distribution within the benchmark. It uses a lightweight two-stage process, ensuring efficiency and fast iterations, while generating benchmarks that reflect the expected traffic. We conduct a thorough line of experiments, showing quantitatively and qualitatively that DataMorgana surpasses existing tools and approaches in producing lexically, syntactically, and semantically diverse question sets across domain-specific and general-knowledge corpora. DataMorgana will be made available to selected teams in the research community, as first beta testers, in the context of the upcoming SIGIR'2025 LiveRAG challenge to be announced in early February 2025.
Paper Structure (15 sections, 3 figures, 5 tables)

This paper contains 15 sections, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. DataMorgana
  4. Configuration Stage
  5. Generation Stage
  6. Experimental Settings
  7. Baselines
  8. Experiments Corpora
  9. Configuration of Question and User Categorizations
  10. Assessing Diversity
  11. Qualitative Diversity Exploration
  12. Quantitative Diversity Evaluation
  13. Diversity Metrics
  14. Experimental Results
  15. Conclusion

Figures (3)

  • Figure 1: DataMorgana generation pipeline.
  • Figure 2: Lexical (NDG, the higher the better - on the left) and syntactic (PoS-CR, the lower the better - on the right) diversity of synthetic benchmarks when increasing the number of questions generated for each of the 147 documents in the COVID-QA dataset. Similar trends are observed with other metrics.
  • Figure 3: Lexical (NDG, the higher the better - on the left) and syntactic (PoS-CR, the lower the better - on the right) diversity of synthetic benchmarks containing 500 questions generated from an increasing number of documents in the COVID-QA dataset. Similar trends are observed with other metrics.