Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Speculative Diffusion Decoding: Accelerating Language Generation through Diffusion

Jacob K Christopher, Brian R Bartoldson, Tal Ben-Nun, Michael Cardei, Bhavya Kailkhura, Ferdinando Fioretto

TL;DR

SpecDiff introduces discrete diffusion language models as drafters in speculative decoding to enable parallel drafting and verification, addressing the sequential bottleneck of autoregressive drafting. By generating entire draft sequences in parallel and using a diffusion-based acceptance mechanism, it achieves up to 7.2x speedups over standard generation and up to 1.75x over existing speculative methods, while maintaining high-quality outputs. The approach leverages MDLM as a drafter and analyzes the trade-offs between diffusion steps and draft length, demonstrating strong performance across CNN/DM, OpenWebText, and MT Bench with reduced FLOPs and memory. However, it faces limitations in diffusion-drafter calibration, tokenizer compatibility, performance on short tasks, and deterministic sampling, pointing to future work in calibration, broader tokenizers, and stochastic generation scenarios.

Abstract

Speculative decoding has emerged as a widely adopted method to accelerate large language model inference without sacrificing the quality of the model outputs. While this technique has facilitated notable speed improvements by enabling parallel sequence verification, its efficiency remains inherently limited by the reliance on incremental token generation in existing draft models. To overcome this limitation, this paper proposes an adaptation of speculative decoding which uses discrete diffusion models to generate draft sequences. This allows parallelization of both the drafting and verification steps, providing significant speedups to the inference process. Our proposed approach, $\textit{Speculative Diffusion Decoding (SpecDiff)}$, is validated on standard language generation benchmarks and empirically demonstrated to provide up to 7.2x speedups over standard generation processes and up to 1.75x speedups over existing speculative decoding approaches.

Speculative Diffusion Decoding: Accelerating Language Generation through Diffusion

TL;DR

SpecDiff introduces discrete diffusion language models as drafters in speculative decoding to enable parallel drafting and verification, addressing the sequential bottleneck of autoregressive drafting. By generating entire draft sequences in parallel and using a diffusion-based acceptance mechanism, it achieves up to 7.2x speedups over standard generation and up to 1.75x over existing speculative methods, while maintaining high-quality outputs. The approach leverages MDLM as a drafter and analyzes the trade-offs between diffusion steps and draft length, demonstrating strong performance across CNN/DM, OpenWebText, and MT Bench with reduced FLOPs and memory. However, it faces limitations in diffusion-drafter calibration, tokenizer compatibility, performance on short tasks, and deterministic sampling, pointing to future work in calibration, broader tokenizers, and stochastic generation scenarios.

Abstract

Speculative decoding has emerged as a widely adopted method to accelerate large language model inference without sacrificing the quality of the model outputs. While this technique has facilitated notable speed improvements by enabling parallel sequence verification, its efficiency remains inherently limited by the reliance on incremental token generation in existing draft models. To overcome this limitation, this paper proposes an adaptation of speculative decoding which uses discrete diffusion models to generate draft sequences. This allows parallelization of both the drafting and verification steps, providing significant speedups to the inference process. Our proposed approach, , is validated on standard language generation benchmarks and empirically demonstrated to provide up to 7.2x speedups over standard generation processes and up to 1.75x speedups over existing speculative decoding approaches.
Paper Structure (32 sections, 2 theorems, 10 equations, 4 figures, 6 tables, 1 algorithm)

This paper contains 32 sections, 2 theorems, 10 equations, 4 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Contributions.
  3. Related Work
  4. Speculative decoding.
  5. Non-autoregressive language models.
  6. Preliminaries and Settings
  7. Speculative decoding.
  8. Speculative Diffusion Models
  9. SpecDiff: Formulation
  10. Drafter's evaluations.
  11. Sequence initialization.
  12. Comparative Analysis of Speculative Decoders
  13. Evaluation
  14. Experimental Setup
  15. Settings.
  16. ...and 17 more sections

Key Result

Theorem A.1

Under standard assumptions, the convergence rate of samplers based on the probability flow Ordinary Differential Equation (ODE), converge at the rate where $d$ is the dimensions of the sample, $\epsilon_\text{score}$ is the error in the score function estimation, $\epsilon_\text{Jacobi}$ is the error in the Jacobian matrices, and universal constant $c_3 > 0$.

Figures (4)

  • Figure 1: Illustration of classical speculative decoding (left) and speculative diffusion decoding (right).
  • Figure 2: Evaluation of SpecDiff's sensitivity to $\gamma$ and number of diffusion steps when optimizing speed (left) and accepted tokens per draft (right) as reported on the OpenWebText task using GPT-2 NEO as the target model. Average token acceptance increases with $T$ (x-axis), as examined theoretically in Appendix \ref{['appendix:theory']}. Despite this, the additional compute required as $T$ is scaled results in reduced speedup.
  • Figure 3: Accepted draft lengths for OpenWebText evaluation.
  • Figure 4: Accepted draft lengths for MT Bench evaluation.

Theorems & Definitions (4)

  • Theorem A.1: li2023towards
  • Definition A.2: leviathan2023fast
  • Corollary A.3: leviathan2023fast
  • proof