Dynamic Depth Decoding: Faster Speculative Decoding for LLMs

TL;DR

Dynamic Depth Decoding (DDD) tackles latency in LLM inference by making the EAGLE-2 draft-depth adaptive to draft-model confidence. It defines a confidence heuristic $H = \log\left(\sum_{i=0}^w \exp(\text{logprobsum}[i])\right)$ over beam logprobs and uses sparse heuristic checks at selected steps to decide whether to continue drafting, with $n=11$ drafts and beam width $w=10$. Empirically, DDD achieves an average speedup of $3.16\times$ over vanilla decoding and provides incremental gains over EAGLE-2, though hardware differences complicate cross-study comparisons. The work demonstrates that dynamically adjusting decoding depth based on draft confidence can substantially reduce latency in speculative decoding, guiding future optimization of draft-model performance and verification efficiency.

Abstract

The acceleration of Large Language Models (LLMs) with speculative decoding provides a significant runtime improvement without any loss of accuracy. Currently, EAGLE-2 is the state-of-the-art speculative decoding method, improving on EAGLE with a dynamic draft tree. We introduce Dynamic Depth Decoding (DDD), which optimises EAGLE-2's tree drafting method using a dynamic depth. This extends the average speedup that EAGLE-2 achieves over EAGLE by $44\%$, giving DDD an average speedup of $3.16$x.

Figures (2)

• Figure 1: Speedup ratio compared to vanilla autoregressive decoding for different EAGLE decoding algorithms on MT-Bench with Temperature=0 on a single NVIDIA A40 GPU.
• Figure 2: EAGLE's decoding tree li2024eagle