Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Re-Rankers as Relevance Judges

Chuan Meng, Jiqun Liu, Mohammad Aliannejadi, Fengran Mo, Jeff Dalton, Maarten de Rijke

TL;DR

This paper investigates whether established re-ranking models can serve as relevance judges, treating relevance judgment as a form of relevance prediction. It reproduces 8 re-rankers from three families and adapts them into binary relevance judges using two strategies: direct generation of a true/false token and score thresholding, evaluated on the TREC-DL 2019–2023 tracks with Cohen’s $\kappa$ for human agreement and Kendall’s $\tau$ for system ordering. Across experiments, re-ranker-based judges outperform the state-of-the-art LLM-based judge UMBRELA in roughly 40%–50% of cases, with Rank1 generally offering the strongest robustness, though gains from scaling are mixed. The study also reveals pronounced biases toward their own and same-family re-rankers and cross-family biases, underscoring the need to mitigate evaluation bias and to leverage mature relevance predictors to reduce duplicated effort; supplementary materials are provided to facilitate reuse and replication.

Abstract

Using large language models (LLMs) to predict relevance judgments has shown promising results. Most studies treat this task as a distinct research line, e.g., focusing on prompt design for predicting relevance labels given a query and passage. However, predicting relevance judgments is essentially a form of relevance prediction, a problem extensively studied in tasks such as re-ranking. Despite this potential overlap, little research has explored reusing or adapting established re-ranking methods to predict relevance judgments, leading to potential resource waste and redundant development. To bridge this gap, we reproduce re-rankers in a re-ranker-as-relevance-judge setup. We design two adaptation strategies: (i) using binary tokens (e.g., "true" and "false") generated by a re-ranker as direct judgments, and (ii) converting continuous re-ranking scores into binary labels via thresholding. We perform extensive experiments on TREC-DL 2019 to 2023 with 8 re-rankers from 3 families, ranging from 220M to 32B, and analyse the evaluation bias exhibited by re-ranker-based judges. Results show that re-ranker-based relevance judges, under both strategies, can outperform UMBRELA, a state-of-the-art LLM-based relevance judge, in around 40% to 50% of the cases; they also exhibit strong self-preference towards their own and same-family re-rankers, as well as cross-family bias.

Re-Rankers as Relevance Judges

TL;DR

This paper investigates whether established re-ranking models can serve as relevance judges, treating relevance judgment as a form of relevance prediction. It reproduces 8 re-rankers from three families and adapts them into binary relevance judges using two strategies: direct generation of a true/false token and score thresholding, evaluated on the TREC-DL 2019–2023 tracks with Cohen’s for human agreement and Kendall’s for system ordering. Across experiments, re-ranker-based judges outperform the state-of-the-art LLM-based judge UMBRELA in roughly 40%–50% of cases, with Rank1 generally offering the strongest robustness, though gains from scaling are mixed. The study also reveals pronounced biases toward their own and same-family re-rankers and cross-family biases, underscoring the need to mitigate evaluation bias and to leverage mature relevance predictors to reduce duplicated effort; supplementary materials are provided to facilitate reuse and replication.

Abstract

Using large language models (LLMs) to predict relevance judgments has shown promising results. Most studies treat this task as a distinct research line, e.g., focusing on prompt design for predicting relevance labels given a query and passage. However, predicting relevance judgments is essentially a form of relevance prediction, a problem extensively studied in tasks such as re-ranking. Despite this potential overlap, little research has explored reusing or adapting established re-ranking methods to predict relevance judgments, leading to potential resource waste and redundant development. To bridge this gap, we reproduce re-rankers in a re-ranker-as-relevance-judge setup. We design two adaptation strategies: (i) using binary tokens (e.g., "true" and "false") generated by a re-ranker as direct judgments, and (ii) converting continuous re-ranking scores into binary labels via thresholding. We perform extensive experiments on TREC-DL 2019 to 2023 with 8 re-rankers from 3 families, ranging from 220M to 32B, and analyse the evaluation bias exhibited by re-ranker-based judges. Results show that re-ranker-based relevance judges, under both strategies, can outperform UMBRELA, a state-of-the-art LLM-based relevance judge, in around 40% to 50% of the cases; they also exhibit strong self-preference towards their own and same-family re-rankers, as well as cross-family bias.
Paper Structure (18 sections, 7 figures, 6 tables)

This paper contains 18 sections, 7 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Task Definition: From Re-ranking to Relevance Judging
  3. Reproducibility Methodology
  4. Research questions and experimental design
  5. Experimental setup
  6. Re-rankers with varying sizes.
  7. Adaptation methods.
  8. Bias analysis: Re-ranker-based judges evaluating re-rankers.
  9. Datasets.
  10. Evaluation metric.
  11. Reference LLM-based judge.
  12. Implementation details.
  13. Results and Discussions
  14. Re-rankers as judges via direct generation
  15. Re-rankers as judges via score thresholding
  16. ...and 3 more sections

Figures (7)

  • Figure 1: RankLLaMA's performance across thresholds on TREC-DL 22 and 23. The best-performing threshold from TREC-DL 22 is used for TREC-DL 23.
  • Figure 2: monoT5's performance across thresholds on TREC-DL 22 and 23.
  • Figure 3: Rank1’s performance across thresholds on TREC-DL 22 and 23.
  • Figure 4: Scatter plots comparing system rankings based on relevance judgments from TREC assessors and monoT5 judges.
  • Figure 5: Scatter plots comparing system rankings based on relevance judgments from TREC assessors and RankLLaMA judges.
  • ...and 2 more figures