Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Do RAG Systems Cover What Matters? Evaluating and Optimizing Responses with Sub-Question Coverage

Kaige Xie, Philippe Laban, Prafulla Kumar Choubey, Caiming Xiong, Chien-Sheng Wu

TL;DR

A novel evaluation framework based on sub-question coverage, which measures how well a RAG system addresses different facets of a question, is introduced, and it is demonstrated that leveraging core sub-questions enhances both retrieval and answer generation in a RAG system, resulting in a 74% win rate over the baseline that lacks sub-questions.

Abstract

Evaluating retrieval-augmented generation (RAG) systems remains challenging, particularly for open-ended questions that lack definitive answers and require coverage of multiple sub-topics. In this paper, we introduce a novel evaluation framework based on sub-question coverage, which measures how well a RAG system addresses different facets of a question. We propose decomposing questions into sub-questions and classifying them into three types -- core, background, and follow-up -- to reflect their roles and importance. Using this categorization, we introduce a fine-grained evaluation protocol that provides insights into the retrieval and generation characteristics of RAG systems, including three commercial generative answer engines: You.com, Perplexity AI, and Bing Chat. Interestingly, we find that while all answer engines cover core sub-questions more often than background or follow-up ones, they still miss around 50% of core sub-questions, revealing clear opportunities for improvement. Further, sub-question coverage metrics prove effective for ranking responses, achieving 82% accuracy compared to human preference annotations. Lastly, we also demonstrate that leveraging core sub-questions enhances both retrieval and answer generation in a RAG system, resulting in a 74% win rate over the baseline that lacks sub-questions.

Do RAG Systems Cover What Matters? Evaluating and Optimizing Responses with Sub-Question Coverage

TL;DR

A novel evaluation framework based on sub-question coverage, which measures how well a RAG system addresses different facets of a question, is introduced, and it is demonstrated that leveraging core sub-questions enhances both retrieval and answer generation in a RAG system, resulting in a 74% win rate over the baseline that lacks sub-questions.

Abstract

Evaluating retrieval-augmented generation (RAG) systems remains challenging, particularly for open-ended questions that lack definitive answers and require coverage of multiple sub-topics. In this paper, we introduce a novel evaluation framework based on sub-question coverage, which measures how well a RAG system addresses different facets of a question. We propose decomposing questions into sub-questions and classifying them into three types -- core, background, and follow-up -- to reflect their roles and importance. Using this categorization, we introduce a fine-grained evaluation protocol that provides insights into the retrieval and generation characteristics of RAG systems, including three commercial generative answer engines: You.com, Perplexity AI, and Bing Chat. Interestingly, we find that while all answer engines cover core sub-questions more often than background or follow-up ones, they still miss around 50% of core sub-questions, revealing clear opportunities for improvement. Further, sub-question coverage metrics prove effective for ranking responses, achieving 82% accuracy compared to human preference annotations. Lastly, we also demonstrate that leveraging core sub-questions enhances both retrieval and answer generation in a RAG system, resulting in a 74% win rate over the baseline that lacks sub-questions.

Paper Structure

This paper contains 16 sections, 1 equation, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. RAG Evaluation
  4. Query Optimization in RAG
  5. Sub-Question Coverage: A New Perspective for RAG Evaluation
  6. Question Decomposition
  7. Human Annotation and Classification Accuracy of Sub-Questions:
  8. Fine-grained Evaluation Protocol
  9. Evaluation of RAG-based Answer Engines
  10. An Automatic Answer Quality Metric
  11. Improving RAG Responses with Core Sub-Question
  12. Experimental Setup:
  13. Conclusions
  14. Limitations
  15. GPT-4 Prompts
  16. ...and 1 more sections

Figures (1)

  • Figure 1: An overview of our RAG evaluation framework based on sub-question coverage. Given a question, RAG's answer, and retrieved chunks, we decompose the question into sub-questions and classify them into three types: core, background, and follow-up; we measure the sub-question coverage rates of both the answer and retrieved chunks with categorized sub-questions and design a fine-grained evaluation protocol to assess three popular RAG-based answer engines (\ref{['sec:evaluation']}). We find the sub-question coverage as an answer quality metric can approximate human perception of answer quality well (\ref{['sec:metric']}). We incorporate core sub-questions into different stages of the RAG workflow and effectively improve its responses (\ref{['sec:core-improve']}).