RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation

TL;DR

RagChecker addresses the challenging problem of evaluating Retrieval-Augmented Generation by offering a fine-grained, claim-entailment framework that diagnoses both retrieval and generation components. It introduces a formal formulation, a dual-predicate design for end-users and developers, and a comprehensive metric suite encompassing overall, retriever, and generator diagnostics. Meta-evaluation against human judgments demonstrates RagChecker achieves stronger correlations than existing metrics, and experiments across 8 RAG systems over 4,162 queries reveal actionable patterns and trade-offs in retrieval quality, model size, and context usage. The framework, benchmark, and findings provide practical guidance for building more faithful, robust RAG systems and are openly released for community use and extension.

Abstract

Despite Retrieval-Augmented Generation (RAG) showing promising capability in leveraging external knowledge, a comprehensive evaluation of RAG systems is still challenging due to the modular nature of RAG, evaluation of long-form responses and reliability of measurements. In this paper, we propose a fine-grained evaluation framework, RAGChecker, that incorporates a suite of diagnostic metrics for both the retrieval and generation modules. Meta evaluation verifies that RAGChecker has significantly better correlations with human judgments than other evaluation metrics. Using RAGChecker, we evaluate 8 RAG systems and conduct an in-depth analysis of their performance, revealing insightful patterns and trade-offs in the design choices of RAG architectures. The metrics of RAGChecker can guide researchers and practitioners in developing more effective RAG systems. This work has been open sourced at https://github.com/amazon-science/RAGChecker.

Figures (10)

• Figure 1: Illustration of the proposed metrics in RagChecker . The upper Venn diagram depicts the comparison between a model response and the ground truth answer, showing possible correct( ), incorrect( ), and missing claims( ). The retrieved chunks are classified into two categories based on the type of claims they contain. Below, we define the overall, retriever, and generator metrics, illustrating how each component of the RAG system is evaluated for its performance.
• Figure 2: The prompt used for converting short answers to long-form answers for the domains of Novel, Finance, Lifestyle, Recreation, Technology, Science, and Writing.
• Figure 3: The human annotation interface and instructions of the meta evaluation dataset.
• Figure 4: Comparison of prediction score distribution between RagChecker and RAGAS Answer Similarity. Each point in the plot represents an instance in the meta evaluation dataset, where the x-axis is the human preference label under corresponding aspect and y-axis is the prediction score of RagChecker and RAGAS Answer Similarity. The distribution of prediction score is represented by the colored area and the dashed line is the mean line.
• Figure 5: The default prompt used for response generation in the main experiments for the 8 RAG baseline systems.