Scaling Laws for Speculative Decoding

TL;DR

This work investigates speculative decoding for large language models through a data-model-batch scaling lens. It uncovers log-linear scaling laws linking acceptance rate to pretraining tokens, draft model capacity, and decoding batch size, and introduces Scylla, a framework that optimally coordinates these dimensions. Empirical results across diverse dense LLMs show consistent gains in acceptance rate and decoding throughput, including up to 2x industrial throughput and notable improvements on summarization and QA tasks. The findings provide a principled approach to accelerating reasoning-intensive decoding while highlighting future directions such as RLHF fine-tuning of draft models and extensions to MoE architectures.

Abstract

The escalating demand for efficient decoding in large language models (LLMs) is particularly critical for reasoning-intensive architectures like OpenAI-o3 and DeepSeek-R1, which depend on extended chain-of-thought reasoning. This study investigates speculative decoding techniques through dense LLM architectures to establish foundational insights for accelerating reasoning tasks. While speculative decoding methods leveraging parallel draft-verification cycles have emerged as promising acceleration techniques, the scaling laws governing decoding efficiency remain under-explored compared to conventional backbone LLMs developed through Pretraining->SFT->RLHF training paradigms. In this work, we discover Log-linear Scaling Laws (Theorem 1.1, 1.2 and 1.3) governing draft model acceptance rate (or decoding speed) across three dimensions: pretraining token volume, draft model capacity, and decoding batch size. Building on these laws, we achieve Scylla, which coordinates multi-dimensional scaling for popular LLMs (Llama2/3, Qwen2.5). Empirical validation shows Scylla achieves 1.5-2.2 higher acceptance rate than EAGLE2 and 0.3 higher than EAGLE3 at temperature T = 0, with peak performance gains on summarization and QA tasks (Figure 2). Industrial inference engine deployments demonstrate 2X decoding throughput improvements over EAGLE2 (Table 5), validating the transformative potential of systematic scaling for efficient LLM inference. Code will be released later.

Figures (8)

• Figure 1: (Left): Scaling law of acceptance rate on MT‑Bench and pretrain tokens; (Middle): Scaling up decoder layers; (Right): Optimal topk‑path and throughput versus decoding batch size.
• Figure 2: Acceptance rate comparison between Scylla and previous SOTA methods across several benchmarks.
• Figure 3: Speculative decoding architecture used for scaling laws.
• Figure 4: (a) Architecture comparison between draft model of EAGLE and Scylla. We enhance Scylla with scaled decoders and output normalization (RMS Norm). (b) Specialized attention mask to efficiently validate candidate sequences.
• Figure 5: Theoretical throughput of Qwen2.5-72B (single-GPU) over batch size.

Theorems & Definitions (3)

• Theorem 1.1: Log-linear Scaling Law of Pretrain tokens
• Theorem 1.2: Log-linear Scaling Law of Draft Model Capacity
• Theorem 1.3: Log-linear Scaling Law of Decoding Batch Size