Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Overcoming Joint Intractability with Lossless Hierarchical Speculative Decoding

Yuxuan Zhou, Fei Huang, Heng Li, Fengyi Wu, Tianyu Wang, Jianwei Zhang, Junyang Lin, Zhi-Qi Cheng

TL;DR

The paper tackles verification bottlenecks in speculative decoding for fast LLM inference by introducing Hierarchical Speculative Decoding (HSD), a provably lossless verification framework that uses a hierarchy of accessible branches to recover the full target distribution while increasing accepted tokens. It combines formal foundations—partial distributions, branch divergence, and hierarchical mass transfer—with a practical capped-branch resampling scheme to achieve a single-step resampling within accessible branches. The authors provide rigorous proofs of losslessness and demonstrate substantial, consistent speedups (averaging around 6-7% BE/DS) across multiple benchmarks and model scales, including notable gains when integrated with EAGLE-3. Empirically, HSD maintains distribution fidelity, shows compatibility with multi-draft setups, and delivers practical efficiency with verification costs minimal relative to forward passes. The work offers a generally applicable, explainable approach that advances decoding efficiency without compromising fidelity, and provides open-source code for adoption and further research.

Abstract

Verification is a key bottleneck in improving inference speed while maintaining distribution fidelity in Speculative Decoding. Recent work has shown that sequence-level verification leads to a higher number of accepted tokens compared to token-wise verification. However, existing solutions often rely on surrogate approximations or are constrained by partial information, struggling with joint intractability. In this work, we propose Hierarchical Speculative Decoding (HSD), a provably lossless verification method that significantly boosts the expected number of accepted tokens and overcomes joint intractability by balancing excess and deficient probability mass across accessible branches. Our extensive large-scale experiments demonstrate that HSD yields consistent improvements in acceptance rates across diverse model families and benchmarks. Moreover, its strong explainability and generality make it readily integrable into a wide range of speculative decoding frameworks. Notably, integrating HSD into EAGLE-3 yields over a 12% performance gain, establishing state-of-the-art decoding efficiency without compromising distribution fidelity. Code is available at https://github.com/ZhouYuxuanYX/Hierarchical-Speculative-Decoding.

Overcoming Joint Intractability with Lossless Hierarchical Speculative Decoding

TL;DR

The paper tackles verification bottlenecks in speculative decoding for fast LLM inference by introducing Hierarchical Speculative Decoding (HSD), a provably lossless verification framework that uses a hierarchy of accessible branches to recover the full target distribution while increasing accepted tokens. It combines formal foundations—partial distributions, branch divergence, and hierarchical mass transfer—with a practical capped-branch resampling scheme to achieve a single-step resampling within accessible branches. The authors provide rigorous proofs of losslessness and demonstrate substantial, consistent speedups (averaging around 6-7% BE/DS) across multiple benchmarks and model scales, including notable gains when integrated with EAGLE-3. Empirically, HSD maintains distribution fidelity, shows compatibility with multi-draft setups, and delivers practical efficiency with verification costs minimal relative to forward passes. The work offers a generally applicable, explainable approach that advances decoding efficiency without compromising fidelity, and provides open-source code for adoption and further research.

Abstract

Verification is a key bottleneck in improving inference speed while maintaining distribution fidelity in Speculative Decoding. Recent work has shown that sequence-level verification leads to a higher number of accepted tokens compared to token-wise verification. However, existing solutions often rely on surrogate approximations or are constrained by partial information, struggling with joint intractability. In this work, we propose Hierarchical Speculative Decoding (HSD), a provably lossless verification method that significantly boosts the expected number of accepted tokens and overcomes joint intractability by balancing excess and deficient probability mass across accessible branches. Our extensive large-scale experiments demonstrate that HSD yields consistent improvements in acceptance rates across diverse model families and benchmarks. Moreover, its strong explainability and generality make it readily integrable into a wide range of speculative decoding frameworks. Notably, integrating HSD into EAGLE-3 yields over a 12% performance gain, establishing state-of-the-art decoding efficiency without compromising distribution fidelity. Code is available at https://github.com/ZhouYuxuanYX/Hierarchical-Speculative-Decoding.
Paper Structure (49 sections, 18 theorems, 126 equations, 4 figures, 8 tables, 3 algorithms)

This paper contains 49 sections, 18 theorems, 126 equations, 4 figures, 8 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Revisiting Tokenwise Speculative Decoding
  4. Accept term.
  5. Resampling term.
  6. Final distribution.
  7. Theoretical Foundations of Hierarchical Speculative Decoding
  8. Recovery of Partial Distributions
  9. Resampling within the Accessible Branch
  10. Resampling in a Hierarchy of Accessible Branches
  11. Hierarchical Speculative Decoding
  12. Naive Hierachicial Speculative Decoding
  13. Hierarchical Speculative Decoding with Capped Branch Resampling
  14. Computational Efficiency
  15. Illustrative Example
  16. ...and 34 more sections

Key Result

Theorem 1

Partial Distribution Recovery. A target distribution over $\Omega'\subseteq \Omega$ can be fully recovered via resampling iff $D_{\Omega'}(p, q)\,\le\;D_{\Omega'}(q, p)$. (See proof in app:proof_partial_recovery.)

Figures (4)

  • Figure 1: Overview of HSD. HSD accepts the draft $\boldsymbol{X}_{\tau}$ by scanning backward from $\gamma$ to $\tau$, and then performs a single resampling at position $\tau+1$ using the corresponding distribution from the resampling hierarchy.
  • Figure 2: GSM8K question with the generated prefix. The text shown is the printed output of the decoded string in Markdown format.
  • Figure : Naive HSD
  • Figure A.1: The average acceptance probability of the entire draft ($\tau=\gamma$) on GSM8K.

Theorems & Definitions (40)

  • Definition 1
  • Theorem 1
  • Definition 2
  • Definition 3
  • Theorem 2
  • Corollary 3
  • Theorem 4
  • Definition 4
  • Definition 5
  • Definition 6
  • ...and 30 more