How important is Recall for Measuring Retrieval Quality?

TL;DR

This work tackles the challenge of evaluating retrieval quality when the total number of relevant documents $N_p$ is unknown in realistic knowledge bases. It compares traditional recall-based metrics ($F$, $nDCG$) with a new recall-free measure $T$ and an estimated variant $F_e$ by correlating them with LLM-based judgments of response quality in a RAG setup using top-$K$ documents. Across datasets with small $N_p$ values and various embeddings, $T$ generally tracks $F$ well and remains robust when $N_p$ is unknown, while $nDCG$ can outperform at high $K/N_p$ in certain datasets; $F_e$ can outperform $F$ when $K/N_p$ is large. The findings offer practical guidance for evaluating retrieval systems in dynamic KBs and highlight the influence of dataset characteristics and $K/N_p$ on metric choice and performance.

Abstract

In realistic retrieval settings with large and evolving knowledge bases, the total number of documents relevant to a query is typically unknown, and recall cannot be computed. In this paper, we evaluate several established strategies for handling this limitation by measuring the correlation between retrieval quality metrics and LLM-based judgments of response quality, where responses are generated from the retrieved documents. We conduct experiments across multiple datasets with a relatively low number of relevant documents (2-15). We also introduce a simple retrieval quality measure that performs well without requiring knowledge of the total number of relevant documents.

Figures (43)

• Figure 1: Obtaining a correlation between a measure of quality of retrieval and LLM judgment score.
• Figure 2: Distribution of the response score (1 to 5) for embedding models shown on Y-axis. On ARXIV; using only segments with minimum 300 samples, the ratio $\frac{K}{N_p}$ is rounded to the first digit.
• Figure 3: Distribution of the response score (1 to 5) for embedding models shown on Y-axis. On MSMARCO; using only segments with minimum 300 samples, the ratio $\frac{K}{N_p}$ is rounded to the first digit.
• Figure 4: Distribution of the response score (1 to 5) for embedding models shown on Y-axis. On HotpotQA-sentences; using only segments with minimum 300 samples, the ratio $\frac{K}{N_p}$ is rounded to the first digit.
• Figure 5: Spearman correlation between the retrieval measures ($T$, $nDCG$ and $F$) and the response score.