Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HeteroSpec: Leveraging Contextual Heterogeneity for Efficient Speculative Decoding

Siran Liu, Yang Ye, Qianchao Zhu, Zane Cao, Yongchao He

TL;DR

HeteroSpec tackles the bottleneck in speculative decoding for large language models by identifying and exploiting contextual heterogeneity in draft-token verification. It introduces a complexity oracle based on Cumulative Meta-Path Top-$K$ Entropy, a data-driven CART-based stratification, and three coordinated optimizations (dynamic drafting depth, Top-$N$ pruning, and just-in-time computation graphs) to allocate verification resources where they yield the most benefit. Across five benchmarks and four open models, HeteroSpec delivers substantial speedups (average around $4.24\times$) while maintaining exact output distributions and incurring minimal overhead, without requiring retraining. The approach is orthogonal to existing draft-model improvements and offers practical scalability for real-world LLM serving by reducing verification cost and enabling adaptive throughput control.

Abstract

Autoregressive decoding inherently limits the inference throughput of Large Language Model (LLM) due to its sequential dependency. Speculative decoding mitigates this by verifying multiple predicted tokens in parallel, but its efficiency remains constrained by what we identify as verification heterogeneity -- the uneven difficulty of verifying different speculative candidates. In practice, a small subset of high-confidence predictions accounts for most successful verifications, yet existing methods treat all candidates uniformly, leading to redundant computation. We present HeteroSpec, a heterogeneity-adaptive speculative decoding framework that allocates verification effort in proportion to candidate uncertainty. HeteroSpec estimates verification complexity using a lightweight entropy-based quantifier, partitions candidates via a data-driven stratification policy, and dynamically tunes speculative depth and pruning thresholds through coordinated optimization. Across five benchmarks and four LLMs, HeteroSpec delivers an average 4.24$\times$ decoding speedup over state-of-the-art methods such as EAGLE-3, while preserving exact output distributions. Crucially, HeteroSpec requires no model retraining and remains compatible with other inference optimizations, making it a practical direction for improving speculative decoding efficiency.

HeteroSpec: Leveraging Contextual Heterogeneity for Efficient Speculative Decoding

TL;DR

HeteroSpec tackles the bottleneck in speculative decoding for large language models by identifying and exploiting contextual heterogeneity in draft-token verification. It introduces a complexity oracle based on Cumulative Meta-Path Top- Entropy, a data-driven CART-based stratification, and three coordinated optimizations (dynamic drafting depth, Top- pruning, and just-in-time computation graphs) to allocate verification resources where they yield the most benefit. Across five benchmarks and four open models, HeteroSpec delivers substantial speedups (average around ) while maintaining exact output distributions and incurring minimal overhead, without requiring retraining. The approach is orthogonal to existing draft-model improvements and offers practical scalability for real-world LLM serving by reducing verification cost and enabling adaptive throughput control.

Abstract

Autoregressive decoding inherently limits the inference throughput of Large Language Model (LLM) due to its sequential dependency. Speculative decoding mitigates this by verifying multiple predicted tokens in parallel, but its efficiency remains constrained by what we identify as verification heterogeneity -- the uneven difficulty of verifying different speculative candidates. In practice, a small subset of high-confidence predictions accounts for most successful verifications, yet existing methods treat all candidates uniformly, leading to redundant computation. We present HeteroSpec, a heterogeneity-adaptive speculative decoding framework that allocates verification effort in proportion to candidate uncertainty. HeteroSpec estimates verification complexity using a lightweight entropy-based quantifier, partitions candidates via a data-driven stratification policy, and dynamically tunes speculative depth and pruning thresholds through coordinated optimization. Across five benchmarks and four LLMs, HeteroSpec delivers an average 4.24 decoding speedup over state-of-the-art methods such as EAGLE-3, while preserving exact output distributions. Crucially, HeteroSpec requires no model retraining and remains compatible with other inference optimizations, making it a practical direction for improving speculative decoding efficiency.
Paper Structure (29 sections, 3 equations, 4 figures, 8 tables)

This paper contains 29 sections, 3 equations, 4 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Speculative Decoding
  4. EAGLEs and its Draft Tree Construction
  5. Observations
  6. The Verification Bottleneck
  7. Characterizing Heterogeneity
  8. Implications for Further Optimization
  9. Methodology
  10. Contextual Complexity Quantification
  11. Adaptive Decision Framework
  12. Coordinated Adaptive Optimization
  13. Experiments
  14. Effectiveness
  15. Ablation Study
  16. ...and 14 more sections

Figures (4)

  • Figure 1: Key empirical observations with EAGLE-3. (a) Illustration of the Terminal Confidence Rank (TCR). (b) Breakdown of runtime overhead during single-turn speculative decoding for models of different sizes. (c) Distribution of Terminal Confidence Rank within the Top-$N$ draft candidates (with prominent values highlighted in orange). (d) Correlation between Average Acceptance Length and Terminal Confidence Rank. The initial rise reflects increasing maximum acceptance length as TCR grows, until reaching the maximum draft depth. (e) Quantile analysis of the Terminal Confidence Rank distribution, showing concentration within the top percentiles of Top-$N$.
  • Figure 2: Illustration of the HeteroSpec framework, where ②, ③, and ④ represent our three unique modules. We demonstrate the main differences between HeteroSpec and EAGLE-3 in the inference pipeline using an example of an EAGLE drafting tree with Top-$K$=2, Top-$N$=20, and Depth=5.
  • Figure 3: Terminal Confidence Rank analysis across multiple models and datasets. Left panels: Distribution of Terminal Confidence Rank within Top-$N$ draft candidates (bar chart, with prominent values highlighted in orange), with inset quantile analysis showing concentration within top percentiles. Right panels: Correlation between Average Acceptance Length and Terminal Confidence Rank (bar chart). (a,b) LLaMA 3.1-8B on HumanEval; (c,d) Vicuna-13B on MT-Bench; (e,f) Vicuna-13B on HumanEval; (g,h) LLaMA 3.3-70B on MT-Bench; (i,j) LLaMA 3.3-70B on HumanEval.
  • Figure 4: Validation results of the proposed Cumulative Meta-Path Top-$K$ Entropy metric across LLaMA 8B, Vicuna 13B, and LLaMA 70B models on MT-Bench and CNN/Daily Mail datasets. For each subplot, the x-axis shows Terminal Confidence Rank (TCR), the left y-axis indicates average acceptance length (bar chart, with prominent values highlighted in orange), and the right y-axis shows average Cumulative Meta-Path Top-$K$ Entropy (line plot).