LLM-Driven Usefulness Judgment for Web Search Evaluation

TL;DR

This work argues that usefulness, not just relevance, is essential for evaluating web search quality. It introduces TRUE, a rubric-based prompting framework that combines implicit and explicit user signals to produce usefulness labels from LLMs, and validates it on THUIR-KDD'19 and TianGong-QRef datasets using a multi-LLM setup with Spearman correlation against human judgments. Through an iterative rubric formation process and targeted ablations, the study shows that TRUE improves usefulness judgments in longer, context-rich sessions and identifies the key signals driving accuracy, while also demonstrating the benefits and limitations of finetuning with LoRA. The results highlight the potential and challenges of scalable, user-centered usefulness evaluation for large-scale search systems, with implications for real-time ranking, evaluation campaigns, and task-driven search experiences.

Abstract

Evaluation is fundamental in optimizing search experiences and supporting diverse user intents in Information Retrieval (IR). Traditional search evaluation methods primarily rely on relevance labels, which assess how well retrieved documents match a user's query. However, relevance alone fails to capture a search system's effectiveness in helping users achieve their search goals, making usefulness a critical evaluation criterion. In this paper, we explore an alternative approach: LLM-generated usefulness labels, which incorporate both implicit and explicit user behavior signals to evaluate document usefulness. We propose Task-aware Rubric-based Usefulness Evaluation (TRUE), a rubric-driven evaluation method that employs iterative sampling and reasoning to model complex search behavior patterns. Our findings show that (i) LLMs can generate moderate usefulness labels by leveraging comprehensive search session history incorporating personalization and contextual understanding, and (ii) fine-tuned LLMs improve usefulness judgments when provided with structured search session contexts. Additionally, we examine whether LLMs can distinguish between relevance and usefulness, particularly in cases where this divergence impacts search success. We also conduct an ablation study to identify key metrics for accurate usefulness label generation, optimizing for token efficiency and cost-effectiveness in real-world applications. This study advances LLM-based usefulness evaluation by refining key user metrics, exploring LLM-generated label reliability, and ensuring feasibility for large-scale search systems.

Figures (11)

• Figure 1: Usefulness label generation with LLMs.
• Figure 2: TRUE formulation.
• Figure 3: Distribution of usefulness labels (human vs LLMs).
• Figure 4: Rubric finetuning on LLaMA-3.1-8B-Instruct (THUIR-KDD'19).
• Figure 5: Rubric finetuning on LLaMA-3.1-8B-Instruct (QRef).