To Judge or not to Judge: Using LLM Judgements for Advertiser Keyphrase Relevance at eBay

TL;DR

This paper addresses aligning advertiser keyphrase relevance with human judgment in a triadic e-commerce ecosystem consisting of seller judgment, Advertising providing keyphrases, and Search auctions. It proposes a distillation approach that uses multiple data sources, including general and fine-tuned LLM judgments, to train small cross-encoders that reconcile buyer/seller signals with search relevance under a business-metrics evaluation framework. The results show that incorporating LLM-derived judgments (especially the LLM(+/-) setup) can substantially reduce the number of keyphrases shown while preserving or improving sales-related metrics and ROAS, and can enhance seller sentiment, illustrating the practical viability of LLM-driven data augmentation and evaluation in large-scale auctions. The study highlights the importance of avoiding biases inherent in click data and emphasizes business-metric grounded evaluation to ensure robust, scalable alignment across the advertising and search systems for a catalog of over 2.3 billion items.

Abstract

E-commerce sellers are recommended keyphrases based on their inventory on which they advertise to increase buyer engagement (clicks/sales). The relevance of advertiser keyphrases plays an important role in preventing the inundation of search systems with numerous irrelevant items that compete for attention in auctions, in addition to maintaining a healthy seller perception. In this work, we describe the shortcomings of training Advertiser keyphrase relevance filter models on click/sales/search relevance signals and the importance of aligning with human judgment, as sellers have the power to adopt or reject said keyphrase recommendations. In this study, we frame Advertiser keyphrase relevance as a complex interaction between 3 dynamical systems -- seller judgment, which influences seller adoption of our product, Advertising, which provides the keyphrases to bid on, and Search, who holds the auctions for the same keyphrases. This study discusses the practicalities of using human judgment via a case study at eBay Advertising and demonstrate that using LLM-as-a-judge en-masse as a scalable proxy for seller judgment to train our relevance models achieves a better harmony across the three systems -- provided that they are bound by a meticulous evaluation framework grounded in business metrics.

Figures (5)

• Figure 1: Screenshot of our keyphrases for manual targeting in Promoted Listings Priority eBay for eBay Advertising.
• Figure 2: A side by side comparison of eBay Advertising for Keyphrase Recommendation funnel and eBay Search funnel for Auctions.
• Figure 3: Auction mechanism of items (Itm) in relation to keyphrases (KP). Red strikethrough font represents filter of Advertising, the underline represents seller curation of keyphrases after advertising has filtered them while gray highlight represents the relevance filter of Search.
• Figure 4: Interface for our human annotators.
• Figure 5: Production Serving architecture for our retrieval and relevance models.