Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Human-AI Comparative Analysis of Prompt Sensitivity in LLM-Based Relevance Judgment

Negar Arabzadeh, Charles L. A . Clarke

TL;DR

The paper tackles prompt sensitivity in LLM-based relevance judgments within information retrieval. It collects 72 prompts from 15 humans and 15 LLMs across binary, graded, and pairwise tasks, and evaluates them on the DL-20/21 datasets using three judges (GPT-4o, LLaMA 3.2-3B, Mistral-7B), with UMBRELA as a baseline and a public data release. Findings show graded relevance is highly sensitive to prompts, while binary and pairwise judgments are more robust; LLM-generated prompts often improve alignment with human labels, especially for simpler tasks, and GPT-4o demonstrates strong cross-prompt robustness. The work provides practical guidance for designing robust LLM-based IR evaluations and a resource for reproducible benchmarking.

Abstract

Large Language Models (LLMs) are increasingly used to automate relevance judgments for information retrieval (IR) tasks, often demonstrating agreement with human labels that approaches inter-human agreement. To assess the robustness and reliability of LLM-based relevance judgments, we systematically investigate impact of prompt sensitivity on the task. We collected prompts for relevance assessment from 15 human experts and 15 LLMs across three tasks~ -- ~binary, graded, and pairwise~ -- ~yielding 90 prompts in total. After filtering out unusable prompts from three humans and three LLMs, we employed the remaining 72 prompts with three different LLMs as judges to label document/query pairs from two TREC Deep Learning Datasets (2020 and 2021). We compare LLM-generated labels with TREC official human labels using Cohen's $κ$ and pairwise agreement measures. In addition to investigating the impact of prompt variations on agreement with human labels, we compare human- and LLM-generated prompts and analyze differences among different LLMs as judges. We also compare human- and LLM-generated prompts with the standard UMBRELA prompt used for relevance assessment by Bing and TREC 2024 Retrieval Augmented Generation (RAG) Track. To support future research in LLM-based evaluation, we release all data and prompts at https://github.com/Narabzad/prompt-sensitivity-relevance-judgements/.

A Human-AI Comparative Analysis of Prompt Sensitivity in LLM-Based Relevance Judgment

TL;DR

The paper tackles prompt sensitivity in LLM-based relevance judgments within information retrieval. It collects 72 prompts from 15 humans and 15 LLMs across binary, graded, and pairwise tasks, and evaluates them on the DL-20/21 datasets using three judges (GPT-4o, LLaMA 3.2-3B, Mistral-7B), with UMBRELA as a baseline and a public data release. Findings show graded relevance is highly sensitive to prompts, while binary and pairwise judgments are more robust; LLM-generated prompts often improve alignment with human labels, especially for simpler tasks, and GPT-4o demonstrates strong cross-prompt robustness. The work provides practical guidance for designing robust LLM-based IR evaluations and a resource for reproducible benchmarking.

Abstract

Large Language Models (LLMs) are increasingly used to automate relevance judgments for information retrieval (IR) tasks, often demonstrating agreement with human labels that approaches inter-human agreement. To assess the robustness and reliability of LLM-based relevance judgments, we systematically investigate impact of prompt sensitivity on the task. We collected prompts for relevance assessment from 15 human experts and 15 LLMs across three tasks~ -- ~binary, graded, and pairwise~ -- ~yielding 90 prompts in total. After filtering out unusable prompts from three humans and three LLMs, we employed the remaining 72 prompts with three different LLMs as judges to label document/query pairs from two TREC Deep Learning Datasets (2020 and 2021). We compare LLM-generated labels with TREC official human labels using Cohen's and pairwise agreement measures. In addition to investigating the impact of prompt variations on agreement with human labels, we compare human- and LLM-generated prompts and analyze differences among different LLMs as judges. We also compare human- and LLM-generated prompts with the standard UMBRELA prompt used for relevance assessment by Bing and TREC 2024 Retrieval Augmented Generation (RAG) Track. To support future research in LLM-based evaluation, we release all data and prompts at https://github.com/Narabzad/prompt-sensitivity-relevance-judgements/.

Paper Structure

This paper contains 8 sections, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Prompt Creation
  3. Prompt generation
  4. Filtering and cleaning
  5. Prompt Diversity
  6. Experimental Methodology
  7. Results and Findings
  8. Conclusion and Limitations

Figures (3)

  • Figure 1: Diversity of words across human and LLM-generated prompts.
  • Figure 2: Agreement of LLM-based relevance judgments with human annotations across different prompts and relevance judgment tasks. UMBRELA represents the reproduction of Bing's LLM assessor introduced in upadhyay2024umbrela. Otherwise, the top 12 bars (H-*) represent human-crafted prompts, while the bottom 12 correspond to LLM-generated prompts. The dashed lines show the mean of agreement in LLM -crafted prompts and human-crafted prompts separately.
  • Figure 3: Krippendorff's inter-agreement rate between all the prompts on two datasets.