Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Redefining Retrieval Evaluation in the Era of LLMs

Giovanni Trappolini, Florin Cuconasu, Simone Filice, Yoelle Maarek, Fabrizio Silvestri

TL;DR

This work argues that traditional IR metrics fail to predict RAG performance because LLMs process retrieved content holistically and are sensitive to distracting passages. It introduces a continuous, LLM-oriented annotation schema that quantifies passage utility and distraction, and defines UDCG, a retrieval evaluation metric that directly correlates with end-to-end answer accuracy. Through experiments on five QA benchmarks and six diverse LLMs, UDCG demonstrates up to 36% higher correlation with RAG performance than conventional metrics and shows robustness to varying context lengths. The approach advances reliable evaluation and optimization of RAG components by aligning metrics with actual downstream generation quality, albeit with limitations related to logit access and language scope.

Abstract

Traditional Information Retrieval (IR) metrics, such as nDCG, MAP, and MRR, assume that human users sequentially examine documents with diminishing attention to lower ranks. This assumption breaks down in Retrieval Augmented Generation (RAG) systems, where search results are consumed by Large Language Models (LLMs), which, unlike humans, process all retrieved documents as a whole rather than sequentially. Additionally, traditional IR metrics do not account for related but irrelevant documents that actively degrade generation quality, rather than merely being ignored. Due to these two major misalignments, namely human vs. machine position discount and human relevance vs. machine utility, classical IR metrics do not accurately predict RAG performance. We introduce a utility-based annotation schema that quantifies both the positive contribution of relevant passages and the negative impact of distracting ones. Building on this foundation, we propose UDCG (Utility and Distraction-aware Cumulative Gain), a metric using an LLM-oriented positional discount to directly optimize the correlation with the end-to-end answer accuracy. Experiments on five datasets and six LLMs demonstrate that UDCG improves correlation by up to 36% compared to traditional metrics. Our work provides a critical step toward aligning IR evaluation with LLM consumers and enables more reliable assessment of RAG components

Redefining Retrieval Evaluation in the Era of LLMs

TL;DR

This work argues that traditional IR metrics fail to predict RAG performance because LLMs process retrieved content holistically and are sensitive to distracting passages. It introduces a continuous, LLM-oriented annotation schema that quantifies passage utility and distraction, and defines UDCG, a retrieval evaluation metric that directly correlates with end-to-end answer accuracy. Through experiments on five QA benchmarks and six diverse LLMs, UDCG demonstrates up to 36% higher correlation with RAG performance than conventional metrics and shows robustness to varying context lengths. The approach advances reliable evaluation and optimization of RAG components by aligning metrics with actual downstream generation quality, albeit with limitations related to logit access and language scope.

Abstract

Traditional Information Retrieval (IR) metrics, such as nDCG, MAP, and MRR, assume that human users sequentially examine documents with diminishing attention to lower ranks. This assumption breaks down in Retrieval Augmented Generation (RAG) systems, where search results are consumed by Large Language Models (LLMs), which, unlike humans, process all retrieved documents as a whole rather than sequentially. Additionally, traditional IR metrics do not account for related but irrelevant documents that actively degrade generation quality, rather than merely being ignored. Due to these two major misalignments, namely human vs. machine position discount and human relevance vs. machine utility, classical IR metrics do not accurately predict RAG performance. We introduce a utility-based annotation schema that quantifies both the positive contribution of relevant passages and the negative impact of distracting ones. Building on this foundation, we propose UDCG (Utility and Distraction-aware Cumulative Gain), a metric using an LLM-oriented positional discount to directly optimize the correlation with the end-to-end answer accuracy. Experiments on five datasets and six LLMs demonstrate that UDCG improves correlation by up to 36% compared to traditional metrics. Our work provides a critical step toward aligning IR evaluation with LLM consumers and enables more reliable assessment of RAG components
Paper Structure (18 sections, 8 equations, 5 figures, 4 tables)

This paper contains 18 sections, 8 equations, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Retrieval Augmented Generation
  4. IR Evaluation for RAG
  5. Flaws of Traditional IR Metrics in RAG
  6. An LLM-oriented Annotation Schema
  7. Defining a RAG-tailored Metric
  8. Experimental Setting
  9. Datasets and Models
  10. Evaluation Strategy
  11. Experimental Results
  12. Ideal Rankings of Different Annotations
  13. Correlation between IR Metrics and Answer Accuracy
  14. Ablation Study for UDCG
  15. Sensitivity to Context Length
  16. ...and 3 more sections

Figures (5)

  • Figure 1: Misalignment between traditional IR metrics (blue) and LLM answer accuracy (red) when a single relevant passage is shifted among irrelevant passages.
  • Figure 2: IR metric robustness across context sizes for Qwen 7B. UDCG maintains a stable correlation with RAG accuracy for all $k$.
  • Figure 3: Prompt for evaluating the utility of a document and for answer generation.
  • Figure 4: Prompt template for document relevance assessment using Claude 3.7 Sonnet as judge.
  • Figure 5: Prompt template for answer correctness assessment using Gemini 2.0 Flash as judge.