Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLMs Can Patch Up Missing Relevance Judgments in Evaluation

Shivani Upadhyay, Ehsan Kamalloo, Jimmy Lin

TL;DR

This paper addresses biases from incomplete relevance judgments in Cranfield-style IR evaluation and proposes an LLM-based framework to label holes with fine-grained relevance labels according to TREC guidelines. The method simulates holes by random removal of gold judgments and patches them using in-context LLM prompts, achieving strong alignment with ground-truth judgments on three TREC DL datasets with Kendall $\tau$ values around $0.87$–$0.92$ for extreme hole scenarios when using Vicuña-7B and GPT-3.5 Turbo. It analyzes zero-shot versus few-shot prompting and includes a data-contamination check to assess leakage risk, showing the approach can outperform simply marking holes as non-relevant. The study offers a practical, open-source tool to patch holes, enabling more reliable evaluation while reducing manual labeling costs.

Abstract

Unjudged documents or holes in information retrieval benchmarks are considered non-relevant in evaluation, yielding no gains in measuring effectiveness. However, these missing judgments may inadvertently introduce biases into the evaluation as their prevalence for a retrieval model is heavily contingent on the pooling process. Thus, filling holes becomes crucial in ensuring reliable and accurate evaluation. Collecting human judgment for all documents is cumbersome and impractical. In this paper, we aim at leveraging large language models (LLMs) to automatically label unjudged documents. Our goal is to instruct an LLM using detailed instructions to assign fine-grained relevance judgments to holes. To this end, we systematically simulate scenarios with varying degrees of holes by randomly dropping relevant documents from the relevance judgment in TREC DL tracks. Our experiments reveal a strong correlation between our LLM-based method and ground-truth relevance judgments. Based on our simulation experiments conducted on three TREC DL datasets, in the extreme scenario of retaining only 10% of judgments, our method achieves a Kendall tau correlation of 0.87 and 0.92 on an average for Vicuña-7B and GPT-3.5 Turbo respectively.

LLMs Can Patch Up Missing Relevance Judgments in Evaluation

TL;DR

This paper addresses biases from incomplete relevance judgments in Cranfield-style IR evaluation and proposes an LLM-based framework to label holes with fine-grained relevance labels according to TREC guidelines. The method simulates holes by random removal of gold judgments and patches them using in-context LLM prompts, achieving strong alignment with ground-truth judgments on three TREC DL datasets with Kendall values around – for extreme hole scenarios when using Vicuña-7B and GPT-3.5 Turbo. It analyzes zero-shot versus few-shot prompting and includes a data-contamination check to assess leakage risk, showing the approach can outperform simply marking holes as non-relevant. The study offers a practical, open-source tool to patch holes, enabling more reliable evaluation while reducing manual labeling costs.

Abstract

Unjudged documents or holes in information retrieval benchmarks are considered non-relevant in evaluation, yielding no gains in measuring effectiveness. However, these missing judgments may inadvertently introduce biases into the evaluation as their prevalence for a retrieval model is heavily contingent on the pooling process. Thus, filling holes becomes crucial in ensuring reliable and accurate evaluation. Collecting human judgment for all documents is cumbersome and impractical. In this paper, we aim at leveraging large language models (LLMs) to automatically label unjudged documents. Our goal is to instruct an LLM using detailed instructions to assign fine-grained relevance judgments to holes. To this end, we systematically simulate scenarios with varying degrees of holes by randomly dropping relevant documents from the relevance judgment in TREC DL tracks. Our experiments reveal a strong correlation between our LLM-based method and ground-truth relevance judgments. Based on our simulation experiments conducted on three TREC DL datasets, in the extreme scenario of retaining only 10% of judgments, our method achieves a Kendall tau correlation of 0.87 and 0.92 on an average for Vicuña-7B and GPT-3.5 Turbo respectively.
Paper Structure (15 sections, 1 equation, 4 figures, 4 tables)

This paper contains 15 sections, 1 equation, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Metrics for Incomplete Judgments:
  4. Automated Relevance Judgment:
  5. LLMs for Text Annotation:
  6. Methodology
  7. Experimental Setup
  8. Data:
  9. Preparing Synthetic Incomplete Judgments:
  10. LLMs Used for Evaluations:
  11. Results and Discussion
  12. Correlation with Complete Judgment:
  13. Zero-Shot vs Few-Shot Prompting:
  14. Data Contamination Testing:
  15. Conclusion

Figures (4)

  • Figure 1: Instructions provided in the prompts (without in-context learning).
  • Figure 2: Examples provided in the prompt for in-context learning.
  • Figure 3: Line plots for comparing Kendall $\tau$ value when the patched incomplete judgment file compared with the complete judgment (ground truth). Three scenarios are being compared for each TREC DL track: holes plugged with GPT-3.5$_\text{Turbo}$ (marked with Blue), Vicuña-7B (marked with Orange), and non-relevant label (marked with Green).
  • Figure 4: Line plots for comparing few-shot (marked with Blue) vs. zero-shot (marked with Orange) prompting for filling up the holes in TREC DL 2019.