Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SAGE: Accelerating Vision-Language Models via Entropy-Guided Adaptive Speculative Decoding

Yujia Tong, Tian Zhang, Yunyang Wan, Kaiwei Lin, Jingling Yuan, Chuang Hu

TL;DR

SAGE tackles the latency of vision-language model inference by replacing fixed speculative-decoding trees with an entropy-guided adaptive strategy. By measuring the draft distribution’s entropy, SAGE dynamically adjusts the speculation tree’s depth and width, leveraging temporal entropy correlations to pre-plan for subsequent steps. The approach delivers substantial speedups—up to 3.36x on dense VLMs and robust gains across MoE and LLM settings—while maintaining exact output equivalence. Theoretical analysis ties entropy to acceptance probability and informs optimal tree configurations, and extensive experiments validate the method across image, video, and language benchmarks with minimal overhead. This work enables more efficient, training-free acceleration of complex multimodal generation.

Abstract

Speculative decoding has emerged as a promising approach to accelerate inference in vision-language models (VLMs) by enabling parallel verification of multiple draft tokens. However, existing methods rely on static tree structures that remain fixed throughout the decoding process, failing to adapt to the varying prediction difficulty across generation steps. This leads to suboptimal acceptance lengths and limited speedup. In this paper, we propose SAGE, a novel framework that dynamically adjusts the speculation tree structure based on real-time prediction uncertainty. Our key insight is that output entropy serves as a natural confidence indicator with strong temporal correlation across decoding steps. SAGE constructs deeper-narrower trees for high-confidence predictions to maximize speculation depth, and shallower-wider trees for uncertain predictions to diversify exploration. SAGE improves acceptance lengths and achieves faster acceleration compared to static tree baselines. Experiments on multiple benchmarks demonstrate the effectiveness of SAGE: without any loss in output quality, it delivers up to $3.36\times$ decoding speedup for LLaVA-OneVision-72B and $3.18\times$ for Qwen2.5-VL-72B.

SAGE: Accelerating Vision-Language Models via Entropy-Guided Adaptive Speculative Decoding

TL;DR

SAGE tackles the latency of vision-language model inference by replacing fixed speculative-decoding trees with an entropy-guided adaptive strategy. By measuring the draft distribution’s entropy, SAGE dynamically adjusts the speculation tree’s depth and width, leveraging temporal entropy correlations to pre-plan for subsequent steps. The approach delivers substantial speedups—up to 3.36x on dense VLMs and robust gains across MoE and LLM settings—while maintaining exact output equivalence. Theoretical analysis ties entropy to acceptance probability and informs optimal tree configurations, and extensive experiments validate the method across image, video, and language benchmarks with minimal overhead. This work enables more efficient, training-free acceleration of complex multimodal generation.

Abstract

Speculative decoding has emerged as a promising approach to accelerate inference in vision-language models (VLMs) by enabling parallel verification of multiple draft tokens. However, existing methods rely on static tree structures that remain fixed throughout the decoding process, failing to adapt to the varying prediction difficulty across generation steps. This leads to suboptimal acceptance lengths and limited speedup. In this paper, we propose SAGE, a novel framework that dynamically adjusts the speculation tree structure based on real-time prediction uncertainty. Our key insight is that output entropy serves as a natural confidence indicator with strong temporal correlation across decoding steps. SAGE constructs deeper-narrower trees for high-confidence predictions to maximize speculation depth, and shallower-wider trees for uncertain predictions to diversify exploration. SAGE improves acceptance lengths and achieves faster acceleration compared to static tree baselines. Experiments on multiple benchmarks demonstrate the effectiveness of SAGE: without any loss in output quality, it delivers up to decoding speedup for LLaVA-OneVision-72B and for Qwen2.5-VL-72B.
Paper Structure (37 sections, 6 theorems, 36 equations, 4 figures, 6 tables, 1 algorithm)

This paper contains 37 sections, 6 theorems, 36 equations, 4 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Vision-Language Models
  4. Speculative Decoding
  5. Methodology
  6. Entropy-Based Confidence Estimation
  7. Adaptive Tree Structure Generation
  8. Adaptive Depth.
  9. Adaptive Width.
  10. Hierarchical Width Decay.
  11. Dynamic Tree Construction.
  12. Temporal Correlation Exploitation.
  13. Overall Inference Pipeline
  14. Phase 1: Tree-based Draft Generation.
  15. Phase 2: Parallel Verification.
  16. ...and 22 more sections

Key Result

Lemma 4.3

Let $\tilde{P}_d$ denote the renormalized top-$k$ distribution with probabilities $p_1 \geq p_2 \geq \cdots \geq p_k$, and let $\alpha = 1 - H(\tilde{P}_d)/\log k$ be the confidence score. Then:

Figures (4)

  • Figure 1: Overview of SAGE. The framework consists of three phases: (1) tree-based draft generation, (2) parallel verification by the target model, and (3) entropy-guided dynamic tree update that computes a confidence score from output entropy and adapts tree depth/width accordingly.
  • Figure 2: Autocorrelation analysis of prediction entropy sequences on Qwen2.5-VL 7B. Left: VideoDetailedCaption dataset. Right: MVBench dataset. Lag-$k$ denotes the correlation between entropy values separated by $k$ decoding steps.
  • Figure 3: Performance comparison between SAGE and SpecVLM across different generation lengths on VideoDetailedCaption (LLaVA-OneVision 72B-7B).
  • Figure 4: Initialized draft tree structure.

Theorems & Definitions (14)

  • Definition 4.1: Acceptance Event
  • Lemma 4.3: Confidence-Probability Relationship
  • Theorem 4.4: Acceptance Probability Lower Bound
  • Theorem 4.6: Expected Acceptance Length
  • Theorem 4.7: Optimal Depth
  • Theorem 4.8: Optimal Width
  • proof : Proof of Lemma \ref{['lem:conf_prob']}
  • proof : Proof of Theorem \ref{['thm:accept_prob']}
  • Remark 1.1
  • Corollary 1.2: Multi-Step Acceptance
  • ...and 4 more