Polybasic Speculative Decoding Through a Theoretical Perspective

TL;DR

This work tackles the latency bottleneck in large language models by introducing polybasic speculative decoding, a multi-drafter framework that extends beyond traditional draft-target setups. It provides a formal problem formulation, a rigorous theory for optimal inference time and stability, and a practical three-model architecture with staged verification. Empirically, the approach delivers up to approximately 4x speedups across multiple models and tasks while preserving the target output distribution, and theory-guided model insertion improves efficiency. The findings offer a principled path to scalable, stable inference and open avenues for further optimization, including caching and distributed implementations.

Abstract

Inference latency stands as a critical bottleneck in the large-scale deployment of Large Language Models (LLMs). Speculative decoding methods have recently shown promise in accelerating inference without compromising the output distribution. However, existing work typically relies on a dualistic draft-verify framework and lacks rigorous theoretical grounding. In this paper, we introduce a novel \emph{polybasic} speculative decoding framework, underpinned by a comprehensive theoretical analysis. Specifically, we prove a fundamental theorem that characterizes the optimal inference time for multi-model speculative decoding systems, shedding light on how to extend beyond the dualistic approach to a more general polybasic paradigm. Through our theoretical investigation of multi-model token generation, we expose and optimize the interplay between model capabilities, acceptance lengths, and overall computational cost. Our framework supports both standalone implementation and integration with existing speculative techniques, leading to accelerated performance in practice. Experimental results across multiple model families demonstrate that our approach yields speedup ratios ranging from $3.31\times$ to $4.01\times$ for LLaMA2-Chat 7B, up to $3.87 \times$ for LLaMA3-8B, up to $4.43 \times$ for Vicuna-7B and up to $3.85 \times$ for Qwen2-7B -- all while preserving the original output distribution. We release our theoretical proofs and implementation code to facilitate further investigation into polybasic speculative decoding.

Figures (4)

• Figure 1: Comparison of speculative decoding frameworks. (a) Traditional dualistic approach with a single draft model. (b) Our polybasic framework with multiple draft models achieves superior performance (4× speedup and 8-10 tokens acceptance length) while maintaining good generalization ability. The framework demonstrates significant improvements over the dualistic baseline.
• Figure 2: Speedup ratios for Vicuna-7B, LLaMA2-Chat 7B, LLaMA3-8B-Instruct and Qwen2-7B-Instruct on SpecBench. Our polybasic system consistently achieves the highest speedups ($3.16\times$--$3.66\times$), surpassing EAGLE2 and vanilla baselines.
• Figure 3: Speedup by task. Our method excels in math tasks, reaching $4.43\times$ with Vicuna-7B, while also maintaining strong accelerations in translation, QA, and multi-turn conversation.
• Figure 4: Variance of acceptance length. Speculative sampling (blue) exhibits noticeably lower variance compared to greedy sampling (orange), aligning with our theoretical stability analysis.

Theorems & Definitions (6)

• Lemma 3.1: Optimal Inference Time
• proof : Sketch of Proof
• Theorem 3.2: Model Insertion Efficiency
• proof
• Theorem 3.3: Sampling Stability
• proof