Large Language Models for Relevance Judgment in Product Search

TL;DR

This study evaluates whether large language models can automate query-item relevance judgments for product search at scale. By finetuning multiple LLMs, including LoRA-based adaptations, on a massive dataset of QIPs with rich item descriptions, the authors demonstrate that carefully configured LLMs can reach or approach human evaluator quality, improving over baseline off-the-shelf models. They show that richer item attributes and appropriate LoRA hyperparameters yield the best results, with larger models benefiting more from description data. The work also validates LLM-based relevance judgments against human judgments in feature evaluation tasks, underscoring potential for scalable, cost-effective relevance labelling in production search systems. Future directions include self-distillation and synthetic labeling to further close the gap to human performance.

Abstract

High relevance of retrieved and re-ranked items to the search query is the cornerstone of successful product search, yet measuring relevance of items to queries is one of the most challenging tasks in product information retrieval, and quality of product search is highly influenced by the precision and scale of available relevance-labelled data. In this paper, we present an array of techniques for leveraging Large Language Models (LLMs) for automating the relevance judgment of query-item pairs (QIPs) at scale. Using a unique dataset of multi-million QIPs, annotated by human evaluators, we test and optimize hyper parameters for finetuning billion-parameter LLMs with and without Low Rank Adaption (LoRA), as well as various modes of item attribute concatenation and prompting in LLM finetuning, and consider trade offs in item attribute inclusion for quality of relevance predictions. We demonstrate considerable improvement over baselines of prior generations of LLMs, as well as off-the-shelf models, towards relevance annotations on par with the human relevance evaluators. Our findings have immediate implications for the growing field of relevance judgment automation in product search.

Figures (1)

• Figure 1: LLM Relevance classification training pipeline