Speculative Decoding: Performance or Illusion?

TL;DR

This work presents the first production-grade, systematic evaluation of speculative decoding (SD) in a widely deployed inference engine (vLLM), benchmarking multiple SD variants across model scales and workloads. It reveals that verification of proposed tokens by the target model remains the primary time cost, while acceptance lengths vary by position, request, and dataset. The authors establish a theoretical upper bound on SD speedup using an oracle-like assumption and demonstrate a notable gap between observed and optimal speedups, and they show that adaptive combinations of SD methods can yield substantial gains. The findings offer practical guidance for deploying SD in real systems and identify concrete research directions to close the gap toward maximal speedups in large-scale inference.

Abstract

Speculative decoding (SD) has become a popular technique to accelerate Large Language Model (LLM) inference, yet its real-world effectiveness remains unclear as prior evaluations rely on research prototypes and unrealistically small batch sizes. We present, to our knowledge, the first systematic study of SD on a production-grade and widely deployed inference engine (vLLM), covering multiple SD variants ($n$-gram, EAGLE/EAGLE-3, Draft-Model, Multi-Token Prediction) across diverse workloads, model scales, and batch sizes. We analyze key factors governing SD performance, and quantify a theoretical upper bound on SD speedup. Our results show that verification by the target model dominates the execution, while acceptance length varies markedly across output token positions, requests, and datasets. Comparing measured performance with theoretical bounds reveals substantial gaps between observed and theoretical upper bounds, and we leverage this observation to highlight new research opportunities that our study opens up in improving SD.

Figures (13)

• Figure 1: End-to-end performance on non-reasoning workloads. For the $n$-gram method, we report results for both three-token and five-token proposals.
• Figure 2: End-to-end performance on reasoning workloads.
• Figure 3: Execution time breakdown across different models.
• Figure 4: Generation length per token position for Llama3.1-8B. Requests are sorted based on generation length. Darker colors indicate that more tokens are generated at the corresponding position.
• Figure 5: Average generation length (tokens) per output token position for Qwen3-8B-Thinking on GPQA-Main. Shaded bands show the 95% confidence interval.