Do LLM-judges Align with Human Relevance in Cranfield-style Recommender Evaluation?

TL;DR

Offline recommender evaluation suffers from incomplete labels and bias; Cranfield-style pooling offers reliability but is costly. The paper investigates using an LLM-judge to predict human relevance judgments within a Cranfield-style framework, using the ML-32M-ext dataset. Results show that richer item metadata and longer user histories improve LLM–human alignment (up to Kendall’s tau 0.87 with human rankings), and the approach scales to industrial settings such as podcast recommendation. The work positions LLM-judge as a scalable, bias-aware complement to offline evaluations, with avenues for human-in-the-loop improvements and bias analysis.

Abstract

Evaluating recommender systems remains a long-standing challenge, as offline methods based on historical user interactions and train-test splits often yield unstable and inconsistent results due to exposure bias, popularity bias, sampled evaluations, and missing-not-at-random patterns. In contrast, textual document retrieval benefits from robust, standardized evaluation via Cranfield-style test collections, which combine pooled relevance judgments with controlled setups. While recent work shows that adapting this methodology to recommender systems is feasible, constructing such collections remains costly due to the need for manual relevance judgments, thus limiting scalability. This paper investigates whether Large Language Models (LLMs) can serve as reliable automatic judges to address these scalability challenges. Using the ML-32M-ext Cranfield-style movie recommendation collection, we first examine the limitations of existing evaluation methodologies. Then we explore the alignment and the recommender systems ranking agreement between the LLM-judge and human provided relevance labels. We find that incorporating richer item metadata and longer user histories improves alignment, and that LLM-judge yields high agreement with human-based rankings (Kendall's tau = 0.87). Finally, an industrial case study in the podcast recommendation domain demonstrates the practical value of LLM-judge for model selection. Overall, our results show that LLM-judge is a viable and scalable approach for evaluating recommender systems.

Figures (8)

• Figure 1: Prompt template, user profile structure, and movie metadata structure used for LLM-judge evaluations.
• Figure 2: Judged@100 by different distribution of train and test size in a historical interaction train-test split.
• Figure 3: Effect on sampling from the full relevance labels from ML-32M-ext in terms of Judged@100 and agreement of the model rankings when using the complete set of labels.
• Figure 4: Effect of increasing the size of the user history on the LLM-judge pair agreement.
• Figure 5: Effect of decreasing the interest level differences between the pairs of labels.