Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RAGVUE: A Diagnostic View for Explainable and Automated Evaluation of Retrieval-Augmented Generation

Keerthana Murugaraj, Salima Lamsiyah, Martin Theobald

TL;DR

RAGVue tackles the challenge of evaluating Retrieval-Augmented Generation by offering a diagnostic, explainable, reference-free framework that decomposes the pipeline into retrieval quality, answer quality, and grounding, augmented by strict claim-level faithfulness and judge calibration. It supports manual and agentic evaluation modes and provides a Python API, CLI, and a local Streamlit UI for seamless integration into research workflows. In StrategyQA-based experiments, RAGVue uncovers fine-grained failures missed by prior tools like RAGAS, demonstrating more actionable diagnostics for debugging and development. The framework reduces reliance on gold references and enables component-level comparisons, aiding practical improvements in real-world RAG systems.

Abstract

Evaluating Retrieval-Augmented Generation (RAG) systems remains a challenging task: existing metrics often collapse heterogeneous behaviors into single scores and provide little insight into whether errors arise from retrieval,reasoning, or grounding. In this paper, we introduce RAGVUE, a diagnostic and explainable framework for automated, reference-free evaluation of RAG pipelines. RAGVUE decomposes RAG behavior into retrieval quality, answer relevance and completeness, strict claim-level faithfulness, and judge calibration. Each metric includes a structured explanation, making the evaluation process transparent. Our framework supports both manual metric selection and fully automated agentic evaluation. It also provides a Python API, CLI, and a local Streamlit interface for interactive usage. In comparative experiments, RAGVUE surfaces fine-grained failures that existing tools such as RAGAS often overlook. We showcase the full RAGVUE workflow and illustrate how it can be integrated into research pipelines and practical RAG development. The source code and detailed instructions on usage are publicly available on GitHub

RAGVUE: A Diagnostic View for Explainable and Automated Evaluation of Retrieval-Augmented Generation

TL;DR

RAGVue tackles the challenge of evaluating Retrieval-Augmented Generation by offering a diagnostic, explainable, reference-free framework that decomposes the pipeline into retrieval quality, answer quality, and grounding, augmented by strict claim-level faithfulness and judge calibration. It supports manual and agentic evaluation modes and provides a Python API, CLI, and a local Streamlit UI for seamless integration into research workflows. In StrategyQA-based experiments, RAGVue uncovers fine-grained failures missed by prior tools like RAGAS, demonstrating more actionable diagnostics for debugging and development. The framework reduces reliance on gold references and enables component-level comparisons, aiding practical improvements in real-world RAG systems.

Abstract

Evaluating Retrieval-Augmented Generation (RAG) systems remains a challenging task: existing metrics often collapse heterogeneous behaviors into single scores and provide little insight into whether errors arise from retrieval,reasoning, or grounding. In this paper, we introduce RAGVUE, a diagnostic and explainable framework for automated, reference-free evaluation of RAG pipelines. RAGVUE decomposes RAG behavior into retrieval quality, answer relevance and completeness, strict claim-level faithfulness, and judge calibration. Each metric includes a structured explanation, making the evaluation process transparent. Our framework supports both manual metric selection and fully automated agentic evaluation. It also provides a Python API, CLI, and a local Streamlit interface for interactive usage. In comparative experiments, RAGVUE surfaces fine-grained failures that existing tools such as RAGAS often overlook. We showcase the full RAGVUE workflow and illustrate how it can be integrated into research pipelines and practical RAG development. The source code and detailed instructions on usage are publicly available on GitHub
Paper Structure (36 sections, 5 equations, 8 figures, 4 tables)

This paper contains 36 sections, 5 equations, 8 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. RAG & Evaluation Challenges.
  4. RAG Evaluation Frameworks & Benchmarks.
  5. RAGVue Framework Overview
  6. RAGVue Metrics
  7. Retrieval Relevance
  8. Retrieval Coverage
  9. Clarity
  10. Answer Relevance
  11. Answer Completeness
  12. Strict Faithfulness
  13. Generic Calibration
  14. Operational Modes & Availability
  15. Operational Modes
  16. ...and 21 more sections

Figures (8)

  • Figure 1: RAGVue Python API usage example.
  • Figure 2: Example usage of the RAGVue command-line interface (ragvue-cli).
  • Figure 3: Example usage of the RAGVue chat Python command-line runner (ragvue-py).
  • Figure 4: Distribution of per-item evaluation time for RAGVue and RAGAS on the 100-query benchmark.
  • Figure 5: RAGVue Streamlit UI: introduction page & overview tab.
  • ...and 3 more figures