DICE: Discrete Interpretable Comparative Evaluation with Probabilistic Scoring for Retrieval-Augmented Generation

TL;DR

DICE tackles trustworthiness in retrieval-augmented generation by coupling an evidence-grounded, two-stage evaluation with probabilistic, confidence-aware A/B/Tie judgments and a scalable Swiss-system ranking. Stage I delivers interpretable reasoning traces grounded in retrieved context, while Stage II converts judgments into quantitative scores using $P = \text{softmax}([l_A, l_B, l_{Tie}])$ and a margin $\Delta P$ to modulate hard vs. soft scoring. The framework scales from $O(N^2)$ to $O(N \log N)$, evidenced by a 42.9% reduction in comparisons, and shows strong alignment with human experts on a Chinese financial QA benchmark, achieving 85.7% accuracy and $\kappa = 0.742$, outperforming a scalar-metric baseline. These results suggest that DICE provides a principled, interpretable, and scalable approach for trustworthy RAG evaluation with practical implications for deployment in high-stakes domains.

Abstract

As Retrieval-Augmented Generation (RAG) systems evolve toward more sophisticated architectures, ensuring their trustworthiness through explainable and robust evaluation becomes critical. Existing scalar metrics suffer from limited interpretability, inadequate uncertainty quantification, and computational inefficiency in multi-system comparisons, hindering responsible deployment of RAG technologies. We introduce DICE (Discrete Interpretable Comparative Evaluation), a two-stage, evidence-coupled framework that advances explainability and robustness in RAG evaluation. DICE combines deep analytical reasoning with probabilistic $\{A, B, Tie\}$ scoring to produce transparent, confidence-aware judgments that support accountable system improvement through interpretable reasoning traces, enabling systematic error diagnosis and actionable insights. To address efficiency challenges at scale, DICE employs a Swiss-system tournament that reduces computational complexity from $O(N^2)$ to $O(N \log N)$, achieving a 42.9% reduction in our eight-system evaluation while preserving ranking fidelity. Validation on a curated Chinese financial QA dataset demonstrates that DICE achieves 85.7% agreement with human experts, substantially outperforming existing LLM-based metrics such as RAGAS. Our results establish DICE as a responsible, explainable, and efficient paradigm for trustworthy RAG system assessment.

Figures (3)

• Figure 1: Overview of the DICE framework: the complete workflow from data preparation to multi-system ranking via Swiss-system tournament and single-system baseline comparison, along with a concrete example of the two-stage evaluation process.
• Figure 2: Confusion matrix detailing alignment patterns between DICE judgments and human expert consensus.
• Figure 3: Final Elo rankings comparing DICE and RAGAS across eight RAG configurations.