Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AdaEAGLE: Optimizing Speculative Decoding via Explicit Modeling of Adaptive Draft Structures

Situo Zhang, Hankun Wang, Da Ma, Zichen Zhu, Lu Chen, Kunyao Lan, Kai Yu

TL;DR

AdaEAGLE proposes explicit adaptive draft-length control for speculative decoding by introducing a Lightweight Draft Length Predictor (LDLP) that estimates the optimal number of draft tokens before generation. Built atop the EAGLE framework, AdaEAGLE continuously aligns draft length with the target model’s evolving state, reducing wasted draft tokens and target forward passes. Empirical results on six benchmarks show that AdaEAGLE achieves a substantial, lossless speedup over vanilla AR decoding (approximately 1.61×) and outperforms fixed-length EAGLE and other adaptive methods, including AdaEAGLE-DDD, with robust performance across tasks. The approach enables deeper optimizations for adaptive draft structures and is easily integrable with existing SD frameworks, potentially extending to tree decoding and non-greedy regimes in future work.

Abstract

Speculative Decoding (SD) is a popular lossless technique for accelerating the inference of Large Language Models (LLMs). We show that the decoding speed of SD frameworks with static draft structures can be significantly improved by incorporating context-aware adaptive draft structures. However, current studies on adaptive draft structures are limited by their performance, modeling approaches, and applicability. In this paper, we introduce AdaEAGLE, the first SD framework that explicitly models adaptive draft structures. AdaEAGLE leverages the Lightweight Draft Length Predictor (LDLP) module to explicitly predict the optimal number of draft tokens during inference to guide the draft model. It achieves comparable speedup results without manual thresholds and allows for deeper, more specialized optimizations. Moreover, together with threshold-based strategies, AdaEAGLE achieves a $1.62\times$ speedup over the vanilla AR decoding and outperforms fixed-length SotA baseline while maintaining output quality.

AdaEAGLE: Optimizing Speculative Decoding via Explicit Modeling of Adaptive Draft Structures

TL;DR

AdaEAGLE proposes explicit adaptive draft-length control for speculative decoding by introducing a Lightweight Draft Length Predictor (LDLP) that estimates the optimal number of draft tokens before generation. Built atop the EAGLE framework, AdaEAGLE continuously aligns draft length with the target model’s evolving state, reducing wasted draft tokens and target forward passes. Empirical results on six benchmarks show that AdaEAGLE achieves a substantial, lossless speedup over vanilla AR decoding (approximately 1.61×) and outperforms fixed-length EAGLE and other adaptive methods, including AdaEAGLE-DDD, with robust performance across tasks. The approach enables deeper optimizations for adaptive draft structures and is easily integrable with existing SD frameworks, potentially extending to tree decoding and non-greedy regimes in future work.

Abstract

Speculative Decoding (SD) is a popular lossless technique for accelerating the inference of Large Language Models (LLMs). We show that the decoding speed of SD frameworks with static draft structures can be significantly improved by incorporating context-aware adaptive draft structures. However, current studies on adaptive draft structures are limited by their performance, modeling approaches, and applicability. In this paper, we introduce AdaEAGLE, the first SD framework that explicitly models adaptive draft structures. AdaEAGLE leverages the Lightweight Draft Length Predictor (LDLP) module to explicitly predict the optimal number of draft tokens during inference to guide the draft model. It achieves comparable speedup results without manual thresholds and allows for deeper, more specialized optimizations. Moreover, together with threshold-based strategies, AdaEAGLE achieves a speedup over the vanilla AR decoding and outperforms fixed-length SotA baseline while maintaining output quality.

Paper Structure

This paper contains 32 sections, 1 theorem, 6 equations, 4 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Analysis of Adaptive Draft Length
  3. Speculative Decoding
  4. EAGLE
  5. Draft-Length Oracle for EAGLE
  6. Benefit of Adaptive Draft Lengths
  7. AdaEAGLE
  8. SD with Adaptive Draft Length
  9. Lightweight Draft Length Predictor
  10. Training of LDLP
  11. Training data collection
  12. Optimization criteria design
  13. Incorporate AdaEAGLE with Other Adaptive Draft Techniques
  14. Experiments
  15. Experimental Setups
  16. ...and 17 more sections

Key Result

Lemma 1

Let the generated formal tokens be $T_{1:j}$. Under EAGLE framework, the optimal draft length is still $k^*_j=\operatorname{optK}(j)$ such that $\hat{t}_{j+k^*_j|j}$ is accepted by the target model, while $\hat{t}_{j+k^*_j+1|j}$ is rejected.

Figures (4)

  • Figure 1: Motivation: finding the optimal draft stopping point to reduce target forward passes and minimize wasted computation.
  • Figure 2: Maximum draft lengths accepted at different draft iterations via EAGLE.
  • Figure 3: Illustration of AdaEAGLE framework. (a) The architecture of AdaEAGLE. The draft length are predicted by an LDLP module. (b) A closer look of LDLP. (c) An example of collecting paired training data of LDLP.
  • Figure 4: Comparison of draft length distributions with and without loss penalty.

Theorems & Definitions (1)

  • Lemma 1