Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FLASH: Latent-Aware Semi-Autoregressive Speculative Decoding for Multimodal Tasks

Zihua Wang, Ruibo Li, Haozhe Du, Joey Tianyi Zhou, Yu Zhang, Xu Yang

TL;DR

FLASH tackles slow inference in large multimodal models by exploiting visual token redundancy and spatial co-occurrence. It introduces a latent-aware visual token compressor and a semi-autoregressive drafting head, trained with a combination of regression and classification objectives to predict multiple tokens per forward pass. Across video captioning and visual instruction tuning tasks, FLASH achieves substantial speed-ups (up to 2.68× on video captioning and 2.55× on visual instruction tuning) while preserving draft quality, demonstrated on LLaVA-1.5 and QwenVL-2.5. This approach enables faster deployment of multimodal models in latency-constrained settings without substantial accuracy loss.

Abstract

Large language and multimodal models (LLMs and LMMs) exhibit strong inference capabilities but are often limited by slow decoding speeds. This challenge is especially acute in LMMs, where visual inputs typically comprise more tokens with lower information density than text -- an issue exacerbated by recent trends toward finer-grained visual tokenizations to boost performance. Speculative decoding has been effective in accelerating LLM inference by using a smaller draft model to generate candidate tokens, which are then selectively verified by the target model, improving speed without sacrificing output quality. While this strategy has been extended to LMMs, existing methods largely overlook the unique properties of visual inputs and depend solely on text-based draft models. In this work, we propose \textbf{FLASH} (Fast Latent-Aware Semi-Autoregressive Heuristics), a speculative decoding framework designed specifically for LMMs, which leverages two key properties of multimodal data to design the draft model. First, to address redundancy in visual tokens, we propose a lightweight latent-aware token compression mechanism. Second, recognizing that visual objects often co-occur within a scene, we employ a semi-autoregressive decoding strategy to generate multiple tokens per forward pass. These innovations accelerate draft decoding while maintaining high acceptance rates, resulting in faster overall inference. Experiments show that FLASH significantly outperforms prior speculative decoding approaches in both unimodal and multimodal settings, achieving up to \textbf{2.68$\times$} speed-up on video captioning and \textbf{2.55$\times$} on visual instruction tuning tasks compared to the original LMM. Our code is available \href{https://github.com/ZihuaEvan/FlashSD/}{[here]}.

FLASH: Latent-Aware Semi-Autoregressive Speculative Decoding for Multimodal Tasks

TL;DR

FLASH tackles slow inference in large multimodal models by exploiting visual token redundancy and spatial co-occurrence. It introduces a latent-aware visual token compressor and a semi-autoregressive drafting head, trained with a combination of regression and classification objectives to predict multiple tokens per forward pass. Across video captioning and visual instruction tuning tasks, FLASH achieves substantial speed-ups (up to 2.68× on video captioning and 2.55× on visual instruction tuning) while preserving draft quality, demonstrated on LLaVA-1.5 and QwenVL-2.5. This approach enables faster deployment of multimodal models in latency-constrained settings without substantial accuracy loss.

Abstract

Large language and multimodal models (LLMs and LMMs) exhibit strong inference capabilities but are often limited by slow decoding speeds. This challenge is especially acute in LMMs, where visual inputs typically comprise more tokens with lower information density than text -- an issue exacerbated by recent trends toward finer-grained visual tokenizations to boost performance. Speculative decoding has been effective in accelerating LLM inference by using a smaller draft model to generate candidate tokens, which are then selectively verified by the target model, improving speed without sacrificing output quality. While this strategy has been extended to LMMs, existing methods largely overlook the unique properties of visual inputs and depend solely on text-based draft models. In this work, we propose \textbf{FLASH} (Fast Latent-Aware Semi-Autoregressive Heuristics), a speculative decoding framework designed specifically for LMMs, which leverages two key properties of multimodal data to design the draft model. First, to address redundancy in visual tokens, we propose a lightweight latent-aware token compression mechanism. Second, recognizing that visual objects often co-occur within a scene, we employ a semi-autoregressive decoding strategy to generate multiple tokens per forward pass. These innovations accelerate draft decoding while maintaining high acceptance rates, resulting in faster overall inference. Experiments show that FLASH significantly outperforms prior speculative decoding approaches in both unimodal and multimodal settings, achieving up to \textbf{2.68} speed-up on video captioning and \textbf{2.55} on visual instruction tuning tasks compared to the original LMM. Our code is available \href{https://github.com/ZihuaEvan/FlashSD/}{[here]}.
Paper Structure (15 sections, 6 equations, 3 figures, 4 tables, 1 algorithm)

This paper contains 15 sections, 6 equations, 3 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. FLASH
  4. Preliminaries
  5. Visual token compression
  6. Semi-autoregressive inference
  7. Experiments
  8. Implementation settings
  9. Results
  10. Video captioning
  11. Visual instruction tuning
  12. Ablations
  13. Conclusion and Limitations
  14. Appendix
  15. Scalability

Figures (3)

  • Figure 1: Illustration of FLASH. The target model is depicted on the left, while the draft model is shown on the right. In this example, the number of the total input tokens $M$ is equal to $N+2$, as the input includes $N$ visual tokens and $2$ textual tokens. The draft model takes the second-to-top layer features $F_V$ as the visual input. These features are first compressed by a visual compressor, producing the compressed visual features $\hat{F}_V$. Along with textual feature and embeddings $\hat{F}_T$, they are fed into a semi-autoregressive head to generate the next $K$ tokens in parallel. These candidate drafts are highlighted with red-bordered boxes.
  • Figure 2: Training procedure with the semi-autoregressive head. The semi-autoregressive head concatenates the compressed visual features $\hat{F}_V$ and the textual features and embeddings $\hat{F}_T$ as input, and subsequently generates the hidden state features $F'$ for the next $K$ tokens. The corresponding probability $P$ for these $K$ candidate tokens are calculated by the frozen LM head from the target model. The semi-autoregressive head is trained using a regression loss, where the feature $F$ from the frozen target model is served as the ground truth, as well as a classification loss, where the draft probability distribution $P'$ is supervised by the output distribution of the target model $P$.
  • Figure 3: Speed-up ratio and average acceptance tokens with different draft length K. The green solid line indicates the speed-up ratio, while the blue dashed line represents the average acceptance tokens.