TRUE: A Reproducible Framework for LLM-Driven Relevance Judgment in Information Retrieval

TL;DR

The paper addresses the lack of reproducibility in LLM-driven relevance judgments for IR by proposing TRUE, a Task-aware Rubric-based Evaluation. TRUE uses a four-phase pipeline to learn and apply rubrics across five relevance features, enabling auditable labeling. Empirical evaluation on DL19, DL20, and LLMJudge shows competitive leaderboard correlations and insights into model-size effects and human-alignment, while highlighting limitations in MAP/MRR due to label noise. The approach advances transparent, reproducible IR evaluation and points to extensions to session-aware and conversational retrieval in the future.

Abstract

LLM-based relevance judgment generation has become a crucial approach in advancing evaluation methodologies in Information Retrieval (IR). It has progressed significantly, often showing high correlation with human judgments as reflected in LLMJudge leaderboards \cite{rahmani2025judging}. However, existing methods for relevance judgments, rely heavily on sensitive prompting strategies, lacking standardized workflows for generating reliable labels. To fill this gap, we reintroduce our method, \textit{Task-aware Rubric-based Evaluation} (TRUE), for relevance judgment generation. Originally developed for usefulness evaluation in search sessions, we extend TRUE to mitigate the gap in relevance judgment due to its demonstrated effectiveness and reproducible workflow. This framework leverages iterative data sampling and reasoning to evaluate relevance judgments across multiple factors including intent, coverage, specificity, accuracy and usefulness. In this paper, we evaluate TRUE on the TREC DL 2019, 2020 and LLMJudge datasets and our results show that TRUE achieves strong performance on the system-ranking LLM leaderboards. The primary focus of this work is to provide a reproducible framework for LLM-based relevance judgments, and we further analyze the effectiveness of TRUE across multiple dimensions.

Figures (2)

• Figure 1: Rubrics formulation with TRUE (Initial 3 steps).
• Figure 2: Comparisons of TRUE across LLMs (first three columns) and between TRUE and TRUE Reasoning on LLMJudge (last two columns). Red = run-level scores and blue = topic/run pairs with rank correlations in terms of Spearman’s ($\rho$).