Fine Grained Evaluation of LLMs-as-Judges

TL;DR

This work interrogates the use of Large Language Models as judges for ad hoc information retrieval, extending beyond document-level relevance to generate and evaluate rationales by highlighting the passages that address a query. Using the INEX Wikipedia collections (2009 and 2010) and few-shot prompts, the authors map LLM-generated rationales to exact document spans and measure precision, recall, and F1 at macro and micro levels, comparing Llama-3.1-8B-Instruct and GPT-4.1-mini across three exemplar configurations. They demonstrate that LLMs can approximate human judgments on relevance but require supervision and careful prompt design; larger models may underperform smaller ones in some settings and tend to over-highlight or struggle with micro-needles in long documents, suggesting that LLMs are not yet ready to replace human assessors. The results emphasize the potential of LLM-based rationales to reduce labeling effort while also outlining challenges such as hallucination, prompt sensitivity, and the need for calibration and model-selection strategies in IR evaluation workflows.

Abstract

A good deal of recent research has focused on how Large Language Models (LLMs) may be used as `judges' in place of humans to evaluate the quality of the output produced by various text / image processing systems. Within this broader context, a number of studies have investigated the specific question of how effectively LLMs can be used as relevance assessors for the standard ad hoc task in Information Retrieval (IR). We extend these studies by looking at additional questions. Most importantly, we use a Wikipedia based test collection created by the INEX initiative, and prompt LLMs to not only judge whether documents are relevant / non-relevant, but to highlight relevant passages in documents that it regards as useful. The human relevance assessors involved in creating this collection were given analogous instructions, i.e., they were asked to highlight all passages within a document that respond to the information need expressed in a query. This enables us to evaluate the quality of LLMs as judges not only at the document level, but to also quantify how often these `judges' are right for the right reasons. Our findings suggest that LLMs-as-judges work best under human supervision.

Figures (7)

• Figure 1: Structure of our prompt. For Example 7, which corresponds to the test instance, the query, narrative and document are provided; the LLM is instructed to provide the relevant excerpts from the document. We highlight the rationale (relevant passage) within the document for the reader's convenience.
• Figure 2: A line from an INEX 2009 qrel file
• Figure 3: Distribution of generated content lengths relative to document lengths across INEX 2009 and 2010 using Exemplar$_2$.
• Figure 4: Distribution of document length vs precision, document length vs fraction of gold chunks, document length vs fraction of generated portions for the INEX 2009 dataset using Exemplar$_2$ with Llama-3.1-8B model. The number of bins was determined using Rice’s Rule ($k = 2n^{1/3}$).
• Figure 5: Distribution of document length vs precision, document length vs fraction of gold chunks, document length vs fraction of generated portions for the INEX 2010 dataset using Exemplar$_1$ with Llama-3.1-8B model. The number of bins was determined using Rice’s Rule ($k = 2n^{1/3}$).