LLM-Evaluation Tropes: Perspectives on the Validity of LLM-Evaluations

TL;DR

The paper interrogates the validity of using large language models as evaluators in information retrieval, highlighting how LLM-derived judgments can mislead if used without safeguards. It develops a taxonomy of evaluation tropes and guardrails, demonstrates risks through case studies and demonstrations of circularity, and proposes a cooperative, ongoing testing framework (coopetition) to maintain validity with fresh data and human benchmarks. The authors argue for careful integration of LLMs as validation tools rather than definitive judgment sources, emphasizing cross-validation, adversarial testing, and transparency. The work contributes practical guidelines for responsible deployment of LLM-based evaluations in IR and outlines a pathway for continual, community-driven verification.

Abstract

Large Language Models (LLMs) are increasingly used to evaluate information retrieval (IR) systems, generating relevance judgments traditionally made by human assessors. Recent empirical studies suggest that LLM-based evaluations often align with human judgments, leading some to suggest that human judges may no longer be necessary, while others highlight concerns about judgment reliability, validity, and long-term impact. As IR systems begin incorporating LLM-generated signals, evaluation outcomes risk becoming self-reinforcing, potentially leading to misleading conclusions. This paper examines scenarios where LLM-evaluators may falsely indicate success, particularly when LLM-based judgments influence both system development and evaluation. We highlight key risks, including bias reinforcement, reproducibility challenges, and inconsistencies in assessment methodologies. To address these concerns, we propose tests to quantify adverse effects, guardrails, and a collaborative framework for constructing reusable test collections that integrate LLM judgments responsibly. By providing perspectives from academia and industry, this work aims to establish best practices for the principled use of LLMs in IR evaluation.

Figures (4)

• Figure 1: LLM-evaluation tropes that can lead to invalid conclusions about evaluation, systems, and the efficacy of human judges that oversee the process. Overarching patterns are circularity $\circlearrowright$, Goodhart's law $\heartsuit$, and loss of variety $\cong$.
• Figure 2: Reranking with an LLM evaluator ( Umbrela) improves performance under human relevance labels. This plot compares the original and reranked versions of all TREC RAG 24 systems based on manual assessment.
• Figure 3: Reproduction of upadhyay2024umbrela: On top 60 original TREC RAG 24 systems and data, the Umbrela LLM evaluator correlates highly with manual assessors. Only few submitted retrieval systems included approaches from LLM evaluators. Each system represents one dot. Red dots mark systems known to contain LLM evaluators clarke2024llm.
• Figure 4: Demonstration of the effects of circularity when using Umbrela as both evluator and ranker using TREC RAG 24 data. Each submitted retrieval system is first re-ranked with Umbrela, then evaluated under NDCG with relevance labels from human judges and the Umbrela evaluator. We see that especially among top ranked systems, the evaluation strategy no longer agrees with human judges on which system is better. Showing same top 60 (of 75) systems as in Figure \ref{['fig:rag24-evaluator']}.