3-Model Speculative Decoding

TL;DR

PyramidSD addresses the latency of large language model inference by introducing a hierarchy of three models—draft ${M_D}$, qualifier ${M_Q}$, and target ${M_T}$—and employing two-stage fuzzy acceptance criteria to bridge the distribution gap between draft and target. By exploiting an entropy gradient across model scales and allowing relaxed divergences with thresholds ${\tau_Q}$ and ${\tau_T}$, PyramidSD enables the use of significantly smaller drafts without sacrificing correctness, and it provides practical, training-free integration into existing pipelines. Empirically, PyramidSD achieves up to ${\sim}1.91\times$ speedups over standard SD (e.g., 124 tokens/s on an RTX 4090) with PSD_A offering robust stability and PSD_F offering higher peak throughput at the cost of greater variance. The approach highlights the value of hierarchical decoding for efficient inference and outlines adaptive and cross-family extensions as fruitful future directions.

Abstract

Speculative Decoding (SD) accelerates inference in large language models by using a smaller draft model to propose tokens, which are then verified by a larger target model. However, the throughput gains of SD are fundamentally limited by a trade-off between draft model size and token acceptance: smaller draft models generate tokens more quickly but exhibit greater divergence from the target model, resulting in lower acceptance rates and reduced speedups. We introduce Pyramid Speculative Decoding (PyramidSD), an extension of SD that inserts an intermediate qualifier model between the draft and target to bridge the distributional gap in output predictions, allowing smaller model to be used for drafting. This hierarchical decoding strategy improves alignment across models, enabling higher acceptance rates and allowing the use of significantly smaller draft models without sacrificing overall performance. PyramidSD builds on fuzzy acceptance criteria to support relaxed divergence thresholds at each stage, improving throughput. In experiments, PyramidSD achieves up to 1.91x generation speed over standard SD, reaching 124 tokens per second on a consumer GPU (RTX 4090). In small-memory settings with a 1B-parameter draft model and an 8B target model, PyramidSD minimally trades target model quality for improved throughput. Overall, PyramidSD offers a practical approach to enhancing speculative decoding efficiency and can be readily applied to existing inference pipelines.

Figures (3)

• Figure 1: PyramidSD leverages the common practice of training family of models sharing a common tokenizer to effectively accelerate decoding speed. $\ell_D$ tokens are drafted by $M_D$ and verified by $M_Q$ until $\ell_Q$ tokens are generated, which are passed to $M_T$ for final verification. As fuzzy threshold compares the logits, $\tau=(\tau_Q,\tau_T)$ can be adjusted to trade quality and acceptance rate $\beta$. $P_{M_D}$ is replaced with $P_{M_Q}$ in target comparison for closer distribution to target model. Example shown has been implemented on instruction-tuned LLaMA 3.2 and 3.1 family of models llama3.
• Figure 2: (Right) Evaluation result comparison on CSQA talmor-etal-2019-commonsenseqa dataset with Llama-3.1-8B, Llama-3.2-3B, and Llama-3.2-1B llama3 Generation speeds for FSD, PSD$_A$, and PSD$_F$ are majorly controlled with $\tau_T$ (FSD, PSD) and $\tau_Q$ (PSD) adjustments. We report the fastest generation speed for each data point. Error bars represent standard deviation for CSQA score. (Left) Generation speed comparison with varying $\tau_T$. Highest speeds are reported.
• Figure 3: Behavior analysis on LLaMA 3.2/3.1 family with 1B, 3B, and 8B models on CSQA dataset. Both entropy and confidence scales with model size, revealing characteristics that can be capitalized for inference speed-up.