Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Generative Product Recommendations for Implicit Superlative Queries

Kaustubh D. Dhole, Nikhita Vedula, Saar Kuzi, Giuseppe Castellucci, Eugene Agichtein, Shervin Malmasi

TL;DR

The paper tackles implicit superlative product queries, where users seek the best items but do not specify attributes explicitly. It introduces SUPERB, a four-level relevance scheme, paired with LLM-based annotations to infer implicit attributes and support multi-criteria ranking. A large synthetic dataset of $35{,}651$ superlative queries is generated from Amazon shopping queries and annotated via four prompting regimes (pointwise, pairwise, listwise, deliberated) to evaluate multiple retrieval pipelines, including BM25, RM3, and LLM-driven re-ranking. Results show that listwise re-ranking typically yields the strongest overall performance, with long-context variants and deliberated prompting offering context-sensitive advantages; BM25 can outperform LLMs on explicit, well-defined criteria. The work provides a practical framework for deploying LLM-assisted ranking in e-commerce, informing when to rely on listwise versus pointwise strategies and how to balance efficiency with annotation quality for high-expectation queries.

Abstract

In Recommender Systems, users often seek the best products through indirect, vague, or under-specified queries, such as "best shoes for trail running". Such queries, also referred to as implicit superlative queries, pose a significant challenge for standard retrieval and ranking systems as they lack an explicit mention of attributes and require identifying and reasoning over complex factors. We investigate how Large Language Models (LLMs) can generate implicit attributes for ranking as well as reason over them to improve product recommendations for such queries. As a first step, we propose a novel four-point schema for annotating the best product candidates for superlative queries called SUPERB, paired with LLM-based product annotations. We then empirically evaluate several existing retrieval and ranking approaches on our new dataset, providing insights and discussing their integration into real-world e-commerce production systems.

Generative Product Recommendations for Implicit Superlative Queries

TL;DR

The paper tackles implicit superlative product queries, where users seek the best items but do not specify attributes explicitly. It introduces SUPERB, a four-level relevance scheme, paired with LLM-based annotations to infer implicit attributes and support multi-criteria ranking. A large synthetic dataset of superlative queries is generated from Amazon shopping queries and annotated via four prompting regimes (pointwise, pairwise, listwise, deliberated) to evaluate multiple retrieval pipelines, including BM25, RM3, and LLM-driven re-ranking. Results show that listwise re-ranking typically yields the strongest overall performance, with long-context variants and deliberated prompting offering context-sensitive advantages; BM25 can outperform LLMs on explicit, well-defined criteria. The work provides a practical framework for deploying LLM-assisted ranking in e-commerce, informing when to rely on listwise versus pointwise strategies and how to balance efficiency with annotation quality for high-expectation queries.

Abstract

In Recommender Systems, users often seek the best products through indirect, vague, or under-specified queries, such as "best shoes for trail running". Such queries, also referred to as implicit superlative queries, pose a significant challenge for standard retrieval and ranking systems as they lack an explicit mention of attributes and require identifying and reasoning over complex factors. We investigate how Large Language Models (LLMs) can generate implicit attributes for ranking as well as reason over them to improve product recommendations for such queries. As a first step, we propose a novel four-point schema for annotating the best product candidates for superlative queries called SUPERB, paired with LLM-based product annotations. We then empirically evaluate several existing retrieval and ranking approaches on our new dataset, providing insights and discussing their integration into real-world e-commerce production systems.

Paper Structure

This paper contains 20 sections, 7 equations, 4 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLMs for Ranking and Recommendation
  4. LLMs for Relevance Labelling
  5. Prompting Approaches
  6. Superlative Search Queries
  7. Implicit Superlative Queries
  8. The SUPERB Relevance Scheme
  9. Dataset Construction
  10. Creation of Superlative Queries
  11. Creating Relevance Annotations
  12. Methods
  13. Results and Analysis
  14. Both BM25 and LLM perform well:
  15. Both BM25 and LLM perform poorly:
  16. ...and 5 more sections

Figures (4)

  • Figure 1: Attributes generated through deliberated prompting for a superlative query.
  • Figure 2: Sample generated label and explanation using the deliberated pointwise approach.
  • Figure 3: Listwise ranking consistently improves the best ranking for different values of K.
  • Figure 4: Listwise scores rank better than BM25 for almost all queries. Moreover, LLMs when employed in a listwise fashion benefit from an initial ranked list as queries with higher BM25 scores tend to get better improvements from the listwise approach.