Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reinforcement Speculative Decoding for Fast Ranking

Yingpeng Du, Tianjun Wei, Zhu Sun, Jie Zhang

TL;DR

This work addresses latency and tail-performance challenges in LLM-based ranking for IR/RS by introducing Reinforcement Speculative Decoding (RSD), an up-to-down decoding paradigm that iteratively refines rankings within a fixed budget $T$. It combines a listwise-aware relevance network with reinforcement learning to learn a ranking-modification policy, using a Spearman-distance-based return and a reference-model baseline to ensure unbiased, low-variance updates. The method avoids fine-tuning the target LLM, reduces decoding cost relative to full autoregressive decoding, and demonstrates strong improvements over STD and SD baselines on MS MARCO, Quora, ML-1M, and Amazon-Games, with detailed ablations supporting the importance of RPO and listwise modeling. The results suggest significant practical impact for real-world ranking systems, though the approach faces challenges in cross-domain generalization and scaling to very large candidate sets.

Abstract

Large Language Models (LLMs) have been widely adopted in ranking systems such as information retrieval (IR) systems and recommender systems (RSs). To alleviate the latency of auto-regressive decoding, some studies explore the single (first) token decoding for ranking approximation, but they suffer from severe degradation in tail positions. Although speculative decoding (SD) methods can be a remedy with verification at different positions, they face challenges in ranking systems due to their left-to-right decoding paradigm. Firstly, ranking systems require strict latency constraints, but verification rounds in SD methods remain agnostic; Secondly, SD methods usually discard listwise ranking knowledge about unaccepted items in previous rounds, hindering future multi-token prediction, especially when candidate tokens are the unaccepted items. In this paper, we propose a Reinforcement Speculative Decoding method for fast ranking inference of LLMs. To meet the ranking systems' latency requirement, we propose an up-to-down decoding paradigm that employs an agent to iteratively modify the ranking sequence under a constrained budget. Specifically, we design a ranking-tailored policy optimization, actively exploring optimal multi-round ranking modification policy verified by LLMs via reinforcement learning (RL). To better approximate the target LLM under the constrained budget, we trigger the agent fully utilizing the listwise ranking knowledge about all items verified by LLMs across different rounds in RL, enhancing the modification policy of the agent. More importantly, we demonstrate the theoretical robustness and advantages of our paradigm and implementation. Experiments on both IR and RS tasks show the effectiveness of our proposed method.

Reinforcement Speculative Decoding for Fast Ranking

TL;DR

This work addresses latency and tail-performance challenges in LLM-based ranking for IR/RS by introducing Reinforcement Speculative Decoding (RSD), an up-to-down decoding paradigm that iteratively refines rankings within a fixed budget . It combines a listwise-aware relevance network with reinforcement learning to learn a ranking-modification policy, using a Spearman-distance-based return and a reference-model baseline to ensure unbiased, low-variance updates. The method avoids fine-tuning the target LLM, reduces decoding cost relative to full autoregressive decoding, and demonstrates strong improvements over STD and SD baselines on MS MARCO, Quora, ML-1M, and Amazon-Games, with detailed ablations supporting the importance of RPO and listwise modeling. The results suggest significant practical impact for real-world ranking systems, though the approach faces challenges in cross-domain generalization and scaling to very large candidate sets.

Abstract

Large Language Models (LLMs) have been widely adopted in ranking systems such as information retrieval (IR) systems and recommender systems (RSs). To alleviate the latency of auto-regressive decoding, some studies explore the single (first) token decoding for ranking approximation, but they suffer from severe degradation in tail positions. Although speculative decoding (SD) methods can be a remedy with verification at different positions, they face challenges in ranking systems due to their left-to-right decoding paradigm. Firstly, ranking systems require strict latency constraints, but verification rounds in SD methods remain agnostic; Secondly, SD methods usually discard listwise ranking knowledge about unaccepted items in previous rounds, hindering future multi-token prediction, especially when candidate tokens are the unaccepted items. In this paper, we propose a Reinforcement Speculative Decoding method for fast ranking inference of LLMs. To meet the ranking systems' latency requirement, we propose an up-to-down decoding paradigm that employs an agent to iteratively modify the ranking sequence under a constrained budget. Specifically, we design a ranking-tailored policy optimization, actively exploring optimal multi-round ranking modification policy verified by LLMs via reinforcement learning (RL). To better approximate the target LLM under the constrained budget, we trigger the agent fully utilizing the listwise ranking knowledge about all items verified by LLMs across different rounds in RL, enhancing the modification policy of the agent. More importantly, we demonstrate the theoretical robustness and advantages of our paradigm and implementation. Experiments on both IR and RS tasks show the effectiveness of our proposed method.

Paper Structure

This paper contains 23 sections, 8 theorems, 31 equations, 5 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminary
  4. Methodology
  5. Up-to-down decoding paradigm
  6. Ranking-tailored policy optimization in RL
  7. Reinforcement speculative decoding method in ranking systems
  8. Relevance network $h_\theta(\cdot)$ for deterministic scores
  9. Selection for advantage $\hat{A}_i$ and reference model $\pi_{\theta_{ref}}(\cdot)$
  10. Training strategy and time complexity analysis.
  11. Experiments
  12. Experimental Setup
  13. Experiment results and analysis
  14. Conclusion and limitation
  15. Prompt for LLMs in IR and RS tasks
  16. ...and 8 more sections

Key Result

Theorem 1

(Monotonicity.) For $i^*_T<K$, we have $i^*_t<i^*_{t+1}$ for all $t<T$.

Figures (5)

  • Figure 1: Comparison among different methods and decoding strategies on MS MARCO dataset with Llama3.2-3B-Instruct, where SD requires 5×LLM encoding.
  • Figure 2: Comparison between existing left-to-right and our up-to-down speculative decoding.
  • Figure 3: Performance of RSD and w/o RA in training process.
  • Figure 4: Investigate on adaptation ability of RSD.
  • Figure 5: Hyper-parameters analysis on (a) KL coefficient $\beta$ and (b) budget.

Theorems & Definitions (13)

  • Theorem 1
  • Theorem 2
  • Theorem 3
  • Theorem 1
  • proof
  • Theorem 2
  • proof
  • Lemma 1
  • proof
  • Lemma 2: Zero-Mean Score Function
  • ...and 3 more