Predicting Retrieval Utility and Answer Quality in Retrieval-Augmented Generation

TL;DR

This work formalizes two prediction tasks in retrieval-augmented generation (RAG): Retrieval Performance Prediction ($\,mathrm{RPP}$) to estimate the utility of retrieved context and Generation Performance Prediction ($\mathrm{GPP}$) to estimate final answer quality, using the utility $U$ defined as the gain in downstream performance from context versus zero-shot generation. It investigates a suite of predictors drawn from three sources—context relevance signals (QPP-based), context and answer perplexity ($\text{PerpC}$, $\text{PerpA}$), and intrinsic document quality measures (e.g., QualT5, readability)—and combines them via a linear-regression ensemble to predict $U$ or $\mathcal{P}(a_k)$. Experiments on Natural Questions with BM25, MonoT5, and E5 retrievers across varying context sizes ($k\in\{2,3,5,7,10\}$) show that ensembles outperform single predictors and that post-generation signals (PerpA) add consistent predictive value, especially for GPP. The results also reveal that RPP is harder than GPP due to dependence on the interaction between retrieved context and the LLM's internal knowledge, highlighting the potential for adaptive RAG pipelines informed by these predictions. Overall, the paper demonstrates that combining topical relevance cues, perplexity-based signals, and document-quality metrics yields accurate estimates of RAG performance and paves the way for dynamic, per-query context optimization in RAG systems.

Abstract

The quality of answers generated by large language models (LLMs) in retrieval-augmented generation (RAG) is largely influenced by the contextual information contained in the retrieved documents. A key challenge for improving RAG is to predict both the utility of retrieved documents -- quantified as the performance gain from using context over generation without context -- and the quality of the final answers in terms of correctness and relevance. In this paper, we define two prediction tasks within RAG. The first is retrieval performance prediction (RPP), which estimates the utility of retrieved documents. The second is generation performance prediction (GPP), which estimates the final answer quality. We hypothesise that in RAG, the topical relevance of retrieved documents correlates with their utility, suggesting that query performance prediction (QPP) approaches can be adapted for RPP and GPP. Beyond these retriever-centric signals, we argue that reader-centric features, such as the LLM's perplexity of the retrieved context conditioned on the input query, can further enhance prediction accuracy for both RPP and GPP. Finally, we propose that features reflecting query-agnostic document quality and readability can also provide useful signals to the predictions. We train linear regression models with the above categories of predictors for both RPP and GPP. Experiments on the Natural Questions (NQ) dataset show that combining predictors from multiple feature categories yields the most accurate estimates of RAG performance.

Figures (3)

• Figure 1: An illustration of how the proposed prediction tasks fit into a standard RAG workflow. Retrieval Performance Prediction (RPP) estimates the gain in answer quality of $k$-shot relative to zero-shot (see Equation \ref{['eq:utility']}). Generation Performance Prediction (GPP) directly estimates the $k$-shot answer quality. Both predictions rely on information extracted from the retrieved documents and the generated answer (as shown in the blue area at the bottom). The grey area denotes components not accessible to any predictors.
• Figure 2: RAG prompt for generating answers for NQ datasets. The first part of an answer unit <answer> is put at the end to prompt the LLM to yield immediate answers.
• Figure 3: Prediction accuracy with varying context size from 2 to 10. Sub-graph (a) shows results for RPP, and (b) for GPP. The retriever is E5 for both settings.