Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Controlling Output Rankings in Generative Engines for LLM-based Search

Haibo Jin, Ruoxi Chen, Peiyan Zhang, Yifeng Luo, Huimin Zeng, Man Luo, Haohan Wang

TL;DR

This work addresses the vulnerability of LLM-based search to upstream retrieval biases by introducing CORE, an optimization framework that controls output rankings through content appended to retrieved items in black-box settings. It presents two solution paradigms—shadow-model and query-based—along with three optimization content types (string, reasoning, and review) and a large-scale benchmark, ProductBench, to evaluate ranking manipulation in realistic product-search scenarios. Across four diverse LLMs, CORE achieves high promotion rates (e.g., PSR@Top-5 ≈ 91%, PSR@Top-3 ≈ 87%, PSR@Top-1 ≈ 80%), demonstrating both effectiveness and limitations under model transfer and defense mechanisms. The work highlights significant implications for visibility, model robustness, and the need for mitigations in LLM-driven search systems, while offering a formal analysis of the ranking dynamics and practical guidance for defense strategies.

Abstract

The way customers search for and choose products is changing with the rise of large language models (LLMs). LLM-based search, or generative engines, provides direct product recommendations to users, rather than traditional online search results that require users to explore options themselves. However, these recommendations are strongly influenced by the initial retrieval order of LLMs, which disadvantages small businesses and independent creators by limiting their visibility. In this work, we propose CORE, an optimization method that \textbf{C}ontrols \textbf{O}utput \textbf{R}ankings in g\textbf{E}nerative Engines for LLM-based search. Since the LLM's interactions with the search engine are black-box, CORE targets the content returned by search engines as the primary means of influencing output rankings. Specifically, CORE optimizes retrieved content by appending strategically designed optimization content to steer the ranking of outputs. We introduce three types of optimization content: string-based, reasoning-based, and review-based, demonstrating their effectiveness in shaping output rankings. To evaluate CORE in realistic settings, we introduce ProductBench, a large-scale benchmark with 15 product categories and 200 products per category, where each product is associated with its top-10 recommendations collected from Amazon's search interface. Extensive experiments on four LLMs with search capabilities (GPT-4o, Gemini-2.5, Claude-4, and Grok-3) demonstrate that CORE achieves an average Promotion Success Rate of \textbf{91.4\% @Top-5}, \textbf{86.6\% @Top-3}, and \textbf{80.3\% @Top-1}, across 15 product categories, outperforming existing ranking manipulation methods while preserving the fluency of optimized content.

Controlling Output Rankings in Generative Engines for LLM-based Search

TL;DR

This work addresses the vulnerability of LLM-based search to upstream retrieval biases by introducing CORE, an optimization framework that controls output rankings through content appended to retrieved items in black-box settings. It presents two solution paradigms—shadow-model and query-based—along with three optimization content types (string, reasoning, and review) and a large-scale benchmark, ProductBench, to evaluate ranking manipulation in realistic product-search scenarios. Across four diverse LLMs, CORE achieves high promotion rates (e.g., PSR@Top-5 ≈ 91%, PSR@Top-3 ≈ 87%, PSR@Top-1 ≈ 80%), demonstrating both effectiveness and limitations under model transfer and defense mechanisms. The work highlights significant implications for visibility, model robustness, and the need for mitigations in LLM-driven search systems, while offering a formal analysis of the ranking dynamics and practical guidance for defense strategies.

Abstract

The way customers search for and choose products is changing with the rise of large language models (LLMs). LLM-based search, or generative engines, provides direct product recommendations to users, rather than traditional online search results that require users to explore options themselves. However, these recommendations are strongly influenced by the initial retrieval order of LLMs, which disadvantages small businesses and independent creators by limiting their visibility. In this work, we propose CORE, an optimization method that \textbf{C}ontrols \textbf{O}utput \textbf{R}ankings in g\textbf{E}nerative Engines for LLM-based search. Since the LLM's interactions with the search engine are black-box, CORE targets the content returned by search engines as the primary means of influencing output rankings. Specifically, CORE optimizes retrieved content by appending strategically designed optimization content to steer the ranking of outputs. We introduce three types of optimization content: string-based, reasoning-based, and review-based, demonstrating their effectiveness in shaping output rankings. To evaluate CORE in realistic settings, we introduce ProductBench, a large-scale benchmark with 15 product categories and 200 products per category, where each product is associated with its top-10 recommendations collected from Amazon's search interface. Extensive experiments on four LLMs with search capabilities (GPT-4o, Gemini-2.5, Claude-4, and Grok-3) demonstrate that CORE achieves an average Promotion Success Rate of \textbf{91.4\% @Top-5}, \textbf{86.6\% @Top-3}, and \textbf{80.3\% @Top-1}, across 15 product categories, outperforming existing ranking manipulation methods while preserving the fluency of optimized content.
Paper Structure (45 sections, 3 theorems, 33 equations, 2 figures, 17 tables)

This paper contains 45 sections, 3 theorems, 33 equations, 2 figures, 17 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Methodology
  4. Threat Model
  5. Problem Definition
  6. Optimization Strategy
  7. Controlling Output Rankings in gEnerative engines
  8. Shadow-Model-Based Output Ranking Control
  9. Query-based Output Ranking Control
  10. ProductBench Construction
  11. Experiments
  12. Experimental Setup
  13. Performance of CORE on ProductBench
  14. Comparison with Existing Ranking Methods
  15. Fluency of Optimization Content
  16. ...and 30 more sections

Key Result

Theorem 3.5

Under Assumptions ass:position-bias--ass:smoothness, consider a target item $i^*$ at initial position $k_0$ with baseline success probability $P_0 = P_\theta[\text{rank}(i^*) = 1] < p_{\text{target}}$. If the learning rate satisfies $\eta = \frac{1}{L}$ and we run iterations, then gradient descent achieves:

Figures (2)

  • Figure 1: Overview of how the LLM processes Alice's query (blue box) and how output ranking is controlled by CORE (orange box), where the Hello Kitty camera, originally the lowest-ranked result, becomes the top-1 item when applied to CORE-optimized content.
  • Figure 2: Overview of CORE. (a) Shadow-model optimization uses a shadow model to approximate the synthesizing LLM and directly compute ranking gradients. (b) Query-based optimization interacts with the LLM through iterative feedback, adjusting item text with reasoning-, and review-based content to guide the target item toward the desired ranking.

Theorems & Definitions (5)

  • Theorem 3.5: Convergence of Shadow Optimization
  • proof
  • Theorem 3.7: Convergence with Model Mismatch
  • proof
  • Corollary 3.8: Performance Gap