Query-Document Dense Vectors for LLM Relevance Judgment Bias Analysis

TL;DR

This work tackles the problem of systematic bias in LLM-based relevance judgments for IR evaluation by moving beyond global agreement to a locality-aware analysis. It introduces a clustering-based framework that embeds joint query–document pairs into a shared semantic space using INSTRUCTOR embeddings, then analyzes agreement via Gwet’s AC1 within and across semantic clusters. A cluster-based agreement variation metric, $\Delta AC1_j(q) = \max_c AC1_j(q,c) - \min_c AC1_j(q,c)$, identifies context-dependent disagreements, while a heuristic flags bias-prone queries and computes a Bias Severity Score to prioritize diagnostically informative cases. Experiments on DL-2019 and DL-2020 show that disagreement concentrates in semantically rich clusters and that definitional, contextual, and policy-related failures drive systematic errors; the framework enables bias-aware IR evaluation and can guide selective human validation to improve efficiency and reliability.

Abstract

Large Language Models (LLMs) have been used as relevance assessors for Information Retrieval (IR) evaluation collection creation due to reduced cost and increased scalability as compared to human assessors. While previous research has looked at the reliability of LLMs as compared to human assessors, in this work, we aim to understand if LLMs make systematic mistakes when judging relevance, rather than just understanding how good they are on average. To this aim, we propose a novel representational method for queries and documents that allows us to analyze relevance label distributions and compare LLM and human labels to identify patterns of disagreement and localize systematic areas of disagreement. We introduce a clustering-based framework that embeds query-document (Q-D) pairs into a joint semantic space, treating relevance as a relational property. Experiments on TREC Deep Learning 2019 and 2020 show that systematic disagreement between humans and LLMs is concentrated in specific semantic clusters rather than distributed randomly. Query-level analyses reveal recurring failures, most often in definition-seeking, policy-related, or ambiguous contexts. Queries with large variation in agreement across their clusters emerge as disagreement hotspots, where LLMs tend to under-recall relevant content or over-include irrelevant material. This framework links global diagnostics with localized clustering to uncover hidden weaknesses in LLM judgments, enabling bias-aware and more reliable IR evaluation.

Figures (5)

• Figure 1: Clustering of query–document (Q–D) embeddings with HDBSCAN. Colored groups ($C_{1}$, $C_{2}$, $C_{3}$) denote dense semantic neighborhoods, while grey points ($C_{-1}$) represent noise. Crosses and stars respectively highlight Q–D pairs for query $q_{1}$ and $q_{2}$, illustrating how documents and queries can fall into different clusters depending on their semantic alignment.
• Figure 2: Agreement scores of Cohen’s $\kappa$ and Gwet’s AC1 for GPT-4o judgments on DL-2019. Each point corresponds to a cluster from the HDBSCAN results.
• Figure 3: Bland–Altman plots for GPT-4o on DL-2019 and DL-2020 under Noise and Non-Noise cluster conditions. Each point represents a query, with the dashed line indicating mean bias and dotted lines showing the limits of agreement (LoA).
• Figure 4: Cluster-Based Agreement Variation $\Delta AC1_{j}(q)$ calculated for all queries
• Figure 5: Top-10 bias-prone queries in DL-2019 and DL-2020 identified by the heuristic