Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Revisiting Human-vs-LLM judgments using the TREC Podcast Track

Watheq Mansour, J. Shane Culpepper, Joel Mackenzie, Andrew Yates

TL;DR

The paper investigates the reliability of LLM-based relevance judgments for IR tasks by re-labeling the TREC Podcast Track data with five LLMs and comparing them to ground-truth human judgments. It analyzes how LLM-human disagreement impacts system rankings and finds that 2020 data show high agreement and ranking stability, while 2021 data exhibit substantial volatility with LLMs favoring lexical systems. A subset of high-disagreement pairs is re-evaluated by senior IR experts, revealing that their judgments align more with LLMs than with the original TREC assessors, challenging the notion of a single perfect human judge. The study highlights the influence of data properties such as podcast transcription noise on LLM assessments and underscores the need for cautious integration of LLM judgments into IR evaluation pipelines.

Abstract

Using large language models (LLMs) to annotate relevance is an increasingly important technique in the information retrieval community. While some studies demonstrate that LLMs can achieve high user agreement with ground truth (human) judgments, other studies have argued for the opposite conclusion. To the best of our knowledge, these studies have primarily focused on classic ad-hoc text search scenarios. In this paper, we conduct an analysis on user agreement between LLM and human experts, and explore the impact disagreement has on system rankings. In contrast to prior studies, we focus on a collection composed of audio files that are transcribed into two-minute segments -- the TREC 2020 and 2021 podcast track. We employ five different LLM models to re-assess all of the query-segment pairs, which were originally annotated by TREC assessors. Furthermore, we re-assess a small subset of pairs where LLM and TREC assessors have the highest disagreement, and found that the human experts tend to agree with LLMs more than with the TREC assessors. Our results reinforce the previous insights of Sormunen in 2002 -- that relying on a single assessor leads to lower user agreement.

Revisiting Human-vs-LLM judgments using the TREC Podcast Track

TL;DR

The paper investigates the reliability of LLM-based relevance judgments for IR tasks by re-labeling the TREC Podcast Track data with five LLMs and comparing them to ground-truth human judgments. It analyzes how LLM-human disagreement impacts system rankings and finds that 2020 data show high agreement and ranking stability, while 2021 data exhibit substantial volatility with LLMs favoring lexical systems. A subset of high-disagreement pairs is re-evaluated by senior IR experts, revealing that their judgments align more with LLMs than with the original TREC assessors, challenging the notion of a single perfect human judge. The study highlights the influence of data properties such as podcast transcription noise on LLM assessments and underscores the need for cautious integration of LLM judgments into IR evaluation pipelines.

Abstract

Using large language models (LLMs) to annotate relevance is an increasingly important technique in the information retrieval community. While some studies demonstrate that LLMs can achieve high user agreement with ground truth (human) judgments, other studies have argued for the opposite conclusion. To the best of our knowledge, these studies have primarily focused on classic ad-hoc text search scenarios. In this paper, we conduct an analysis on user agreement between LLM and human experts, and explore the impact disagreement has on system rankings. In contrast to prior studies, we focus on a collection composed of audio files that are transcribed into two-minute segments -- the TREC 2020 and 2021 podcast track. We employ five different LLM models to re-assess all of the query-segment pairs, which were originally annotated by TREC assessors. Furthermore, we re-assess a small subset of pairs where LLM and TREC assessors have the highest disagreement, and found that the human experts tend to agree with LLMs more than with the TREC assessors. Our results reinforce the previous insights of Sormunen in 2002 -- that relying on a single assessor leads to lower user agreement.
Paper Structure (7 sections, 1 figure, 2 tables)

This paper contains 7 sections, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Experimental Setup
  4. Results
  5. Conclusion
  6. Acknowledgments.
  7. Disclosure of Interests.

Figures (1)

  • Figure 1: Order volatility in TREC systems scored using RBP $\phi = 0.95$ in 2020 (left) and 2021 (right), when ordering changes using LLM assessments. The $y$-axis expresses the difference in rank position compared to the ground truth system ordering.