Variations in Relevance Judgments and the Shelf Life of Test Collections

TL;DR

This work re-examines Cranfield-style offline evaluation in the neural IR era by re-annotating the DL'19 test collection with additional annotators to assess how relevance disagreement and narrative absence affect system rankings. It confirms that, despite low inter-annotator agreement under four-grade relevance, system order remains highly correlated with the original judgments, though some modern rankers—particularly LLM-based and distilled models—show degraded performance under new judgments. The study also explores the notion of test collection expiration, identifying an empirical ceiling near $nDCG@10=0.81$ relative to human judges and highlighting the risk of overfitting when iterating on benchmarks. Overall, the authors advocate cautious use of test collections for benchmark-driven progress, emphasize the need for reproducible releases of judgments, and discuss implications for the design and retirement of neural IR benchmarks.

Abstract

The fundamental property of Cranfield-style evaluations, that system rankings are stable even when assessors disagree on individual relevance decisions, was validated on traditional test collections. However, the paradigm shift towards neural retrieval models affected the characteristics of modern test collections, e.g., documents are short, judged with four grades of relevance, and information needs have no descriptions or narratives. Under these changes, it is unclear whether assessor disagreement remains negligible for system comparisons. We investigate this aspect under the additional condition that the few modern test collections are heavily re-used. Given more possible query interpretations due to less formalized information needs, an ``expiration date'' for test collections might be needed if top-effectiveness requires overfitting to a single interpretation of relevance. We run a reproducibility study and re-annotate the relevance judgments of the 2019~TREC Deep Learning track. We can reproduce prior work in the neural retrieval setting, showing that assessor disagreement does not affect system rankings. However, we observe that some models substantially degrade with our new relevance judgments, and some have already reached the effectiveness of humans as rankers, providing evidence that test collections can expire.

Figures (5)

• Figure 1: Transitions in relevance grades from primary to secondary annotations.
• Figure 2: Comparing 3 queries of varying difficulty by inter-annotator agreement under fixed narratives (2 narratives (1) and (2)). Annotators were divided into two groups and were assigned identical document pools. Color indicates strength of agreement; redder indicates stronger agreement, bluer indicates weaker agreement.
• Figure 3: Comparing the stability of judgments through nDCG@10 and system swap probabilities over 10000 judgment combinations. We filter lexical comparisons to improve visibility in the central mass. 'LLM-Neural' denotes that a pair contains one LLM-based and one Neural-based ranker.
• Figure 4: Variance in nDCG@10 over annotator combinations. Effectiveness values on new annotations are translucent. Color of points indicates model type.
• Figure 5: Aggregated system rank changes over all natural combinations of secondary versus primary annotations measured using nDCG@10. Negative $\Delta$ indicates a system frequently improves in rank.