Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Entropy-Aware Speculative Decoding Toward Improved LLM Reasoning

Tiancheng Su, Meicong Zhang, Guoxiu He

TL;DR

The paper addresses the bottleneck of latency in autoregressive LLMs by enhancing speculative decoding with a training-free, entropy-aware penalty. By monitoring entropy in both the draft and target models and applying a dynamic penalty when high uncertainty and substantial overlap occur, EASD reduces error propagation at the token level without sacrificing speed. Empirical results across diverse reasoning benchmarks show that EASD outperforms standard SD and reward-guided variants, and can even surpass the inherent performance of the target model in many cases. The approach offers a practical, low-cost improvement for efficient LLM reasoning with broad applicability to existing SD pipelines.

Abstract

Speculative decoding (SD) accelerates large language model (LLM) reasoning by using a small draft model to generate candidate tokens, which the target LLM either accepts directly or regenerates upon rejection. However, excessive alignment between the draft and target models constrains SD to the performance of the target LLM. To address this limitation, we propose Entropy-Aware Speculative Decoding (EASD), a training-free enhancement. Building on standard SD, EASD incorporates a dynamic entropy-based penalty. At each decoding step, we employ the entropy of the sampling distribution to quantify model uncertainty. When both models exhibit high entropy with substantial overlap among their top-N predictions, the corresponding token is rejected and re-sampled by the target LLM. This penalty prevents low-confidence errors from propagating. By incorporating draft-model verification, EASD enables the possibility of surpassing the target model's inherent performance. Experiments across multiple reasoning benchmarks demonstrate that EASD consistently outperforms existing SD methods and, in most cases, surpasses the target LLM itself. We further prove that the efficiency of EASD is comparable to that of SD. The code can be found in the Supplementary Materials.

Entropy-Aware Speculative Decoding Toward Improved LLM Reasoning

TL;DR

The paper addresses the bottleneck of latency in autoregressive LLMs by enhancing speculative decoding with a training-free, entropy-aware penalty. By monitoring entropy in both the draft and target models and applying a dynamic penalty when high uncertainty and substantial overlap occur, EASD reduces error propagation at the token level without sacrificing speed. Empirical results across diverse reasoning benchmarks show that EASD outperforms standard SD and reward-guided variants, and can even surpass the inherent performance of the target model in many cases. The approach offers a practical, low-cost improvement for efficient LLM reasoning with broad applicability to existing SD pipelines.

Abstract

Speculative decoding (SD) accelerates large language model (LLM) reasoning by using a small draft model to generate candidate tokens, which the target LLM either accepts directly or regenerates upon rejection. However, excessive alignment between the draft and target models constrains SD to the performance of the target LLM. To address this limitation, we propose Entropy-Aware Speculative Decoding (EASD), a training-free enhancement. Building on standard SD, EASD incorporates a dynamic entropy-based penalty. At each decoding step, we employ the entropy of the sampling distribution to quantify model uncertainty. When both models exhibit high entropy with substantial overlap among their top-N predictions, the corresponding token is rejected and re-sampled by the target LLM. This penalty prevents low-confidence errors from propagating. By incorporating draft-model verification, EASD enables the possibility of surpassing the target model's inherent performance. Experiments across multiple reasoning benchmarks demonstrate that EASD consistently outperforms existing SD methods and, in most cases, surpasses the target LLM itself. We further prove that the efficiency of EASD is comparable to that of SD. The code can be found in the Supplementary Materials.
Paper Structure (27 sections, 9 equations, 4 figures, 7 tables, 1 algorithm)

This paper contains 27 sections, 9 equations, 4 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Collaboration Between Large and Small Language Models
  4. Speculative Decoding
  5. Entropy-Based Inference
  6. Entropy-Aware Speculative Decoding
  7. Formulation
  8. Overview
  9. Entropy Computation
  10. Triggering Conditions for Dynamic Penalty
  11. Application of the Dynamic Penalty
  12. Threshold Selection
  13. Experiments
  14. Experiments Settings
  15. Results Analysis
  16. ...and 12 more sections

Figures (4)

  • Figure 1: Comparison of traditional SD, the state-of-the-art method RSD, and our proposed EASD. Traditional SD enforces strict alignment with the target model, while RSD selectively accepts outputs from the draft model. However, when the target model itself produces suboptimal tokens, neither SD nor RSD can correct or redirect the output. EASD overcomes this limitation by introducing token-level adjustments. For example, when both the draft and target models exhibit high entropy on the token The, EASD rejects it and chooses This instead. This change enables a new generation path, This is a classic ‘stars and bars’ problem, instead of directly enumerating the answer as in the original output.
  • Figure 2: Success Case: Answer comparison on a GPQA reasoning problem, illustrating the difference between the original output and the EASD-enhanced output.
  • Figure 3: Success Case: Answer comparison on a Math reasoning problem, illustrating the difference between the original output and the EASD-output.
  • Figure 4: Bad Case: Answer comparison on a Math reasoning problem, illustrating the difference between the original output and the EASD-output.