LLM-based relevance assessment still can't replace human relevance assessment

TL;DR

This paper critically evaluates the claim that LLM-based relevance assessments can replace human judgments in IR. It reproduces Umbrela results, showing a strong overall correlation (e.g., Kendall's $\tau$ around $0.84$–$0.89$) but reveals that the alignment weakens among top-scoring systems and can be exploited or distorted. Through empirical demonstrations and simulations, it uncovers risks of subversion of automatic evaluation, the circularity of using LLM-based judgments to re-rank while also evaluating with the same method, and the emergence of biases that prevent LLM judgments from serving as gold standards. The authors argue that due to biases, Narcissistic tendencies of LLMs, and potential degradation of future LLM performance, automatic judgments are not a safe substitute for human judgments, urging caution and safeguards in future IR evaluation practices.

Abstract

The use of large language models (LLMs) for relevance assessment in information retrieval has gained significant attention, with recent studies suggesting that LLM-based judgments provide comparable evaluations to human judgments. Notably, based on TREC 2024 data, Upadhyay et al make a bold claim that LLM-based relevance assessments, such as those generated by the Umbrela system, can fully replace traditional human relevance assessments in TREC-style evaluations. This paper critically examines this claim, highlighting practical and theoretical limitations that undermine the validity of this conclusion. First, we question whether the evidence provided by Upadhyay et al. genuinely supports their claim, particularly when the test collection is intended to serve as a benchmark for future research innovations.Second, we submit a system deliberately crafted to exploit automatic evaluation metrics, demonstrating that it can achieve artificially inflated scores without truly improving retrieval quality. Third, we simulate the consequences of circularity by analyzing Kendall's tau correlations under the hypothetical scenario in which all systems adopt Umbrela as a final-stage re-ranker, illustrating how reliance on LLM-based assessments can distort system rankings. Theoretical challenges - including the inherent narcissism of LLMs, the risk of overfitting to LLM-based metrics, and the potential degradation of future LLM performance - that must be addressed before LLM-based relevance assessments can be considered a viable replacement for human judgments.

Figures (4)

• Figure 1: Scatter plot extracted from Figure 3 of upadhyay2024largescalestudyrelevanceassessments comparing manual assessment and automatic assessment. Each red dot represents the average performance of a run over all queries. The blue dots plot all run-query combination. The inset provides a closer view of the points in the green rectangle.
• Figure 2: Reproduction of upadhyay2024largescalestudyrelevanceassessments: On top 60 original TREC RAG 24 systems and data, the Umbrela LLM evaluator correlates highly with manual assessors. Only few submitted retrieval systems included approaches from LLM evaluators. Each system represents one dot. Red dots mark systems provided by team WaterlooClarke, which are known to contain LLM evaluations for re-ranking.
• Figure 3: Reranking with an LLM evaluator ( Umbrela) improves performance under human relevance labels. This plot compares the original and reranked versions of all TREC RAG 24 systems based on manual assessment.
• Figure 4: Demonstration of the effects of circularity when using Umbrela as both evluator and ranker using TREC RAG 24 data. Each submitted retrieval system is first re-ranked with Umbrela, then evaluated under NDCG with relevance labels from human judges and the Umbrela evaluator. We see that especially among top ranked systems, the evaluation strategy no longer agrees with human judges on which system is better. Axis ranges are adjusted to display the same top 60 (of 75) systems as in Figure \ref{['fig:rag24-evaluator']}.