Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DREAM: Drafting with Refined Target Features and Entropy-Adaptive Cross-Attention Fusion for Multimodal Speculative Decoding

Yunhai Hu, Tianhua Xia, Zining Liu, Rahul Raman, Xingyu Liu, Bo Bao, Eric Sather, Vithursan Thangarasa, Sai Qian Zhang

TL;DR

DREAM tackles slow autoregressive decoding in vision-language models by introducing a cross-attention-based mechanism that injects target-model intermediate features into a draft model, an entropy-driven adaptive selection of these features for supervision, and a visual-token compression scheme to cut draft latency. The approach yields up to $3.6\times$ speedups across diverse VLMs while maintaining high speculative draft acceptance, demonstrating robust gains on tasks ranging from structured QA to segmentation. Key contributions include a cross-attention fusion module, a dynamic intermediate-feature distillation strategy based on attention entropy, and a data-driven visual token reduction technique, all integrated into a practical training and inference pipeline. The results support DREAM as a scalable method for fast multimodal inference, with publicly available code and clear limitations to address in future work.

Abstract

Speculative decoding (SD) has emerged as a powerful method for accelerating autoregressive generation in large language models (LLMs), yet its integration into vision-language models (VLMs) remains underexplored. We introduce DREAM, a novel speculative decoding framework tailored for VLMs that combines three key innovations: (1) a cross-attention-based mechanism to inject intermediate features from the target model into the draft model for improved alignment, (2) adaptive intermediate feature selection based on attention entropy to guide efficient draft model training, and (3) visual token compression to reduce draft model latency. DREAM enables efficient, accurate, and parallel multimodal decoding with significant throughput improvement. Experiments across a diverse set of recent popular VLMs, including LLaVA, Pixtral, SmolVLM and Gemma3, demonstrate up to 3.6x speedup over conventional decoding and significantly outperform prior SD baselines in both inference throughput and speculative draft acceptance length across a broad range of multimodal benchmarks. The code is publicly available at: https://github.com/SAI-Lab-NYU/DREAM.git

DREAM: Drafting with Refined Target Features and Entropy-Adaptive Cross-Attention Fusion for Multimodal Speculative Decoding

TL;DR

DREAM tackles slow autoregressive decoding in vision-language models by introducing a cross-attention-based mechanism that injects target-model intermediate features into a draft model, an entropy-driven adaptive selection of these features for supervision, and a visual-token compression scheme to cut draft latency. The approach yields up to speedups across diverse VLMs while maintaining high speculative draft acceptance, demonstrating robust gains on tasks ranging from structured QA to segmentation. Key contributions include a cross-attention fusion module, a dynamic intermediate-feature distillation strategy based on attention entropy, and a data-driven visual token reduction technique, all integrated into a practical training and inference pipeline. The results support DREAM as a scalable method for fast multimodal inference, with publicly available code and clear limitations to address in future work.

Abstract

Speculative decoding (SD) has emerged as a powerful method for accelerating autoregressive generation in large language models (LLMs), yet its integration into vision-language models (VLMs) remains underexplored. We introduce DREAM, a novel speculative decoding framework tailored for VLMs that combines three key innovations: (1) a cross-attention-based mechanism to inject intermediate features from the target model into the draft model for improved alignment, (2) adaptive intermediate feature selection based on attention entropy to guide efficient draft model training, and (3) visual token compression to reduce draft model latency. DREAM enables efficient, accurate, and parallel multimodal decoding with significant throughput improvement. Experiments across a diverse set of recent popular VLMs, including LLaVA, Pixtral, SmolVLM and Gemma3, demonstrate up to 3.6x speedup over conventional decoding and significantly outperform prior SD baselines in both inference throughput and speculative draft acceptance length across a broad range of multimodal benchmarks. The code is publicly available at: https://github.com/SAI-Lab-NYU/DREAM.git

Paper Structure

This paper contains 19 sections, 4 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Speculative Decoding
  4. Vision Language Model and Computation Profiling
  5. Intermediate Feature Distillation
  6. Methodology
  7. Cross-attention Mechanism for Efficient Knowledge Injection
  8. Adaptive Intermediate Feature Selection for Draft Training
  9. Loss Functions
  10. Visual Token Compression
  11. Empirical Results
  12. Evaluation Results
  13. Ablation Study
  14. Impact of Draft Model Architecture
  15. Impact of Intermediate Feature Selection
  16. ...and 4 more sections

Figures (4)

  • Figure 1: (a) Standard VLM. (b) DREAM overview.
  • Figure 2: Computational cost of VLMs processing text only (Txt) and multi-modal (Img+Txt) inputs.
  • Figure 3: (a) illustrates the training paradigm of DREAM, while (b) and (c) depict its inference workflow. For simplicity, the tree decoding is not shown in (b) and (c).
  • Figure 4: Normalized speedup S and normalized accepted token length $\tau$ across (a) intermediate feature selection strategies. (b) chain-based and tree-based decoding. (c) visual token compression ratios, where 1 and 3/4 denote $100\%$ and $75\%$ of the visual tokens are retained, respectively. (d) loss weight settings, where the number is the value for $\lambda_{feat}$ and $\lambda_{intermed}$. $\lambda_{KL}$ is fixed to 1.