Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TABED: Test-Time Adaptive Ensemble Drafting for Robust Speculative Decoding in LVLMs

Minjae Lee, Wonjun Kang, Byeongkeun Ahn, Christian Classen, Kevin Galim, Seunghyuk Oh, Minghao Yan, Hyung Il Koo, Kangwook Lee

TL;DR

This work tackles the challenge of accelerating inference in large vision-language models (LVLMs) with speculative decoding (SD). It identifies that existing drafting methods fail to maintain robust speedups across diverse real-world LVLM scenarios. The authors propose Test-time Adaptive Batched Ensemble Drafting (TABED), a training-free, plug-and-play framework that ensembles multiple drafts obtained via batched inference and dynamically weights them using ground-truth feedback from SD verification. TABED achieves a robust average walltime speedup of $1.74\times$ and over $5\%$ improvement compared to single drafting methods, while remaining computationally inexpensive and compatible with extended SD techniques such as token-tree verification and alternative drafting schemes (caption and pooling). The work further analyzes dynamic weighting behavior and demonstrates extensibility through additional drafting candidates, offering a practical path to robust LVLM inference acceleration in diverse, real-world settings.

Abstract

Speculative decoding (SD) has proven effective for accelerating LLM inference by quickly generating draft tokens and verifying them in parallel. However, SD remains largely unexplored for Large Vision-Language Models (LVLMs), which extend LLMs to process both image and text prompts. To address this gap, we benchmark existing inference methods with small draft models on 11 datasets across diverse input scenarios and observe scenario-specific performance fluctuations. Motivated by these findings, we propose Test-time Adaptive Batched Ensemble Drafting (TABED), which dynamically ensembles multiple drafts obtained via batch inference by leveraging deviations from past ground truths available in the SD setting. The dynamic ensemble method achieves an average robust walltime speedup of 1.74x over autoregressive decoding and a 5% improvement over single drafting methods, while remaining training-free and keeping ensembling costs negligible through parameter sharing. With its plug-and-play compatibility, we further enhance TABED by integrating advanced verification and alternative drafting methods. Code and custom-trained models are available at https://github.com/furiosa-ai/TABED.

TABED: Test-Time Adaptive Ensemble Drafting for Robust Speculative Decoding in LVLMs

TL;DR

This work tackles the challenge of accelerating inference in large vision-language models (LVLMs) with speculative decoding (SD). It identifies that existing drafting methods fail to maintain robust speedups across diverse real-world LVLM scenarios. The authors propose Test-time Adaptive Batched Ensemble Drafting (TABED), a training-free, plug-and-play framework that ensembles multiple drafts obtained via batched inference and dynamically weights them using ground-truth feedback from SD verification. TABED achieves a robust average walltime speedup of and over improvement compared to single drafting methods, while remaining computationally inexpensive and compatible with extended SD techniques such as token-tree verification and alternative drafting schemes (caption and pooling). The work further analyzes dynamic weighting behavior and demonstrates extensibility through additional drafting candidates, offering a practical path to robust LVLM inference acceleration in diverse, real-world settings.

Abstract

Speculative decoding (SD) has proven effective for accelerating LLM inference by quickly generating draft tokens and verifying them in parallel. However, SD remains largely unexplored for Large Vision-Language Models (LVLMs), which extend LLMs to process both image and text prompts. To address this gap, we benchmark existing inference methods with small draft models on 11 datasets across diverse input scenarios and observe scenario-specific performance fluctuations. Motivated by these findings, we propose Test-time Adaptive Batched Ensemble Drafting (TABED), which dynamically ensembles multiple drafts obtained via batch inference by leveraging deviations from past ground truths available in the SD setting. The dynamic ensemble method achieves an average robust walltime speedup of 1.74x over autoregressive decoding and a 5% improvement over single drafting methods, while remaining training-free and keeping ensembling costs negligible through parameter sharing. With its plug-and-play compatibility, we further enhance TABED by integrating advanced verification and alternative drafting methods. Code and custom-trained models are available at https://github.com/furiosa-ai/TABED.
Paper Structure (87 sections, 3 equations, 9 figures, 22 tables, 1 algorithm)

This paper contains 87 sections, 3 equations, 9 figures, 22 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Large Vision Language Models
  4. LVLMs
  5. Approximate Inference
  6. Speculative Decoding
  7. SD for LLMs
  8. SD for LVLMs
  9. Test-time Adaptation
  10. Preliminaries: Speculative Decoding
  11. Algorithm
  12. Block Efficiency and Walltime Speedup
  13. Benchmarking SD for LVLMs
  14. Experiment Settings
  15. Target and Draft Models
  16. ...and 72 more sections

Figures (9)

  • Figure 1: Overview of TABED. LVLMs must handle diverse input scenarios involving various combinations of turn-taking, tasks, and potential noise. To effectively accelerate target LVLMs with SD, different drafting methods are required for intra- and inter-response cases. For example, in the first turn, the drafting method needs to identify the visual context referenced by the user based on the input image and then perform text-based knowledge retrieval (① → ②). In the second turn, it performs mathematical reasoning using the accumulated text context (③). Across such varying scenarios, existing single drafting methods with small models—whether multimodal or text-only—exhibit fluctuating performance (\ref{['sec:study']}). TABED addresses this by dynamically ensembling multiple drafts using past ground truths in SD, achieving robust speedups across diverse scenarios (\ref{['sec:ensemble']}). It can be further enhanced by integrating advanced verification and drafting methods (\ref{['sec:further']}). See \ref{['fig:tta_ibed']} for details.
  • Figure 2: TABED predicts the optimal weight, $w^*$, based on the deviation (i.e., distance) of past drafting blocks from the ground truth obtained through the verification in SD. It then dynamically ensembles multiple drafts obtained via batch inference. See Algorithm \ref{['ensemblealgo']} for further details.
  • Figure 3: Qualitative samples visualizing dynamic ensemble weights across datasets. The x-axis and y-axis represent decoding steps and the proportion of $w^{(M)}$ relative to $w^{(M)} + w^{(T)}$, respectively. The acceptance range (shaded) comprises weights under which $M_p{}$ accepts the drafted tokens, while the rejection range (unshaded) comprises weights under which it rejects them. See \ref{['app:quali_w']} for more examples.
  • Figure 4: Additional qualitative samples of dynamic ensemble weights
  • Figure 5: Empirical demonstration of Remarks 2 and 3.
  • ...and 4 more figures

Theorems & Definitions (3)

  • Remark 1
  • Remark 2
  • Remark 3