Evaluating the Utility of Grounding Documents with Reference-Free LLM-based Metrics

TL;DR

Grounding utility in Retrieval Augmented Generation faces challenges with annotation-heavy metrics and LLM-agnostic relevance. The authors introduce GroGU, a model-specific, reference-free metric defined as $ \text{GroGU}_\theta(q,\mathbf{D_r}) = \gamma(y_g \mid q, \mathbf{D_r}) - \gamma(y_u \mid q) $, to quantify how grounding documents affect generation confidence, along with a robust variant KeyEntropy that focuses on salient tokens. They demonstrate GroGU captures utility beyond mere relevance and can differentiate utility across models and layouts, enabling annotation-free data curation for training a query-rewriter with Direct Preference Optimization (DPO). Applying GroGU to select high-utility rewrites yields consistent improvements in retrieval performance (up to 18.2 in MRR) and downstream generation accuracy (up to 9.4 percentage points) across sparse and dense retrievers, highlighting practical impact for tuning RAG systems without labeled data.

Abstract

Retrieval Augmented Generation (RAG)'s success depends on the utility the LLM derives from the content used for grounding. Quantifying content utility does not have a definitive specification and existing metrics ignore model-specific capabilities and/or rely on costly annotations. In this paper, we propose Grounding Generation Utility (GroGU), a model-specific and reference-free metric that defines utility as a function of the downstream LLM's generation confidence based on entropy. Despite having no annotation requirements, GroGU is largely faithful in distinguishing ground-truth documents while capturing nuances ignored by LLM-agnostic metrics. We apply GroGU to train a query-rewriter for RAG by identifying high-utility preference data for Direct Preference Optimization. Experiments show improvements by up to 18.2 points in Mean Reciprocal Rank and up to 9.4 points in answer accuracy.

Figures (5)

• Figure 1: The same document may have different LLM-specific utility. The weaker model (Qwen-2-1.5-it) cannot utilize this document likely due to its challenging format. Both models are sampled 50 times under Qwen's default sampling configuration, with consistent results.
• Figure 2: Procedure to test the utility of the same document for different LLMs.
• Figure 3: Query-rewriting illustration from jangIterCQRIterativeConversational2024. The user's queries are dependent on the conversation history and need to be reformulated into stand-alone queries for the retriever.
• Figure 4: Example showing that non-answer tokens can skew the uncertainty/confidence measure. The model answers the question correctly when grounded on the gold passage but the average entropy is higher, indicating a lower confidence. This is because the incorrect answer repeats part of the question which skews down the average entropy of the generation.
• Figure 5: Example showing that KeyEntropy mitigates the issue of non-answer tokens. The model answers the question correctly when grounded on the gold passage but the average entropy is higher, indicating a lower confidence. KeyEntropy identifies and omits the tokens in the incorrect answer that repeat part of the question, which skews the entropy calculation.