RAGExplorer: A Visual Analytics System for the Comparative Diagnosis of RAG Systems

TL;DR

RAGExplorer addresses the challenge that Retrieval-Augmented Generation performance emerges from a complex, multi-component configuration space. It proposes a visual analytics workflow that moves from a holistic performance overview to interactive, hypothesis-driven, instance-level debugging, supported by quantitative metrics, hierarchical failure attribution, and comparative causal verification. The system comprises four coordinated views and enables side-by-side configuration comparisons to reveal causal failure patterns and context-driven explanations. Case studies and expert evaluations demonstrate that the approach uncovers non-obvious trade-offs and supports data-driven design decisions, highlighting a path toward robust, human-AI collaborative optimization of modular RAG systems.

Abstract

The advent of Retrieval-Augmented Generation (RAG) has significantly enhanced the ability of Large Language Models (LLMs) to produce factually accurate and up-to-date responses. However, the performance of a RAG system is not determined by a single component but emerges from a complex interplay of modular choices, such as embedding models and retrieval algorithms. This creates a vast and often opaque configuration space, making it challenging for developers to understand performance trade-offs and identify optimal designs. To address this challenge, we present RAGExplorer, a visual analytics system for the systematic comparison and diagnosis of RAG configurations. RAGExplorer guides users through a seamless macro-to-micro analytical workflow. Initially, it empowers developers to survey the performance landscape across numerous configurations, allowing for a high-level understanding of which design choices are most effective. For a deeper analysis, the system enables users to drill down into individual failure cases, investigate how differences in retrieved information contribute to errors, and interactively test hypotheses by manipulating the provided context to observe the resulting impact on the generated answer. We demonstrate the effectiveness of RAGExplorer through detailed case studies and user studies, validating its ability to empower developers in navigating the complex RAG design space. Our code and user guide are publicly available at https://github.com/Thymezzz/RAGExplorer.

Figures (6)

• Figure 1: RAGExplorer helps users optimize RAG pipelines through four main stages. After a user (A) defines a configuration space, the system generates (B) a performance overview based on (E) retrieval metrics (Recall, MRR, MAP) or Accuracy. Based on this overview, users can (C) select two configurations to compare, for which (F) an automated algorithm provides a hierarchical attribution of their failure points. This allows users to (D) analyze the context similarity distribution. They can also (G) modify the context to regenerate answers and verify the impact of a specific instance.
• Figure 2: A hierarchical algorithm for diagnosing RAG failure points. It applies a prioritized cascade of checks to assign a single, primary failure point to each incorrect answer.
• Figure 3: This figure illustrates question ID.9 with retrieved chunk 336 and chunk 7949. Irrelevant text is grayed out, supporting_sentences are underlined in blue, and evidence is highlighted in orange. These stylings may be combined where the text passages overlap.
• Figure 4: This case study illustrates how an expert resolves a conflict about overlap using our system. (A–B) In the Performance Overview of Accuracy and MRR, all overlap settings (0, 100, 200) appear identical, suggesting that overlap is irrelevant. (C–D) Guided by the Failure Attribution View, the expert finds that overlap=100 and 200 behave similarly, but overlap=0 shows a distinct failure pattern—fewer FP2 (missed top-ranked documents) yet more FP3 (not in context). (E) The Instance Diagnosis Comparison of case Q.348 reveals why: overlap=0 retrieves more evidence into the rerank range, though not ranked in top-k. This uncovers a benefit and suggests the next step: adding a reranker.
• Figure 5: The Performance Overview results for the expert's initial "stronger-is-better" hypothesis. The view shows the "Strongest" configuration (Config A: emb-8B + reranker-8B + chunk size=2000) achieves 55% accuracy, while the "Budget" configuration (Config B: emb-0.6B + reranker-0.6B + chunk size=2000) achieves 59% accuracy.