MASSV: Multimodal Adaptation and Self-Data Distillation for Speculative Decoding of Vision-Language Models

TL;DR

MASSV first connects the target VLM's vision encoder to the draft model via a lightweight trainable projector, then applies self-distilled visual instruction tuning using responses generated by the target VLM to align token predictions, providing a scalable, architecture-compatible method for accelerating both current and future VLMs.

Abstract

Speculative decoding significantly accelerates language model inference by enabling a lightweight draft model to propose multiple tokens that a larger target model verifies simultaneously. However, applying this technique to vision-language models (VLMs) presents two fundamental challenges: small language models that could serve as efficient drafters lack the architectural components to process visual inputs, and their token predictions fail to match those of VLM target models that consider visual context. We introduce Multimodal Adaptation and Self-Data Distillation for Speculative Decoding of Vision-Language Models (MASSV), which transforms existing small language models into effective multimodal drafters through a two-phase approach. MASSV first connects the target VLM's vision encoder to the draft model via a lightweight trainable projector, then applies self-distilled visual instruction tuning using responses generated by the target VLM to align token predictions. Comprehensive experiments across the Qwen2.5-VL and Gemma3 model families demonstrate that MASSV increases accepted length by up to 30% and delivers end-to-end inference speedups of up to 1.46x on visually-grounded tasks. MASSV provides a scalable, architecture-compatible method for accelerating both current and future VLMs.

Figures (5)

• Figure 1: End-to-end wallclock speedups when drafting for Qwen2.5-VL 7B Instruct at temperature $T = 0$ with speculation length $\gamma = 5$. The baseline uses Qwen2.5-1.5B as a text-only drafter (image tokens removed). MASSV consistently yields the highest speedups across all categories, achieving up to 1.46$\times$ on COCO captioning and 1.28$\times$ overall. The gains are most pronounced for visually grounded tasks, demonstrating the importance of multimodal adaptation and self-distilled visual instruction for accelerating VLM inference.
• Figure 2: Detailed architecture of MASSV illustrating: (1) the two-phase training methodology consisting of multimodal projector pretraining followed by self-distilled visual instruction tuning, and (2) the deployment configuration for draft token generation during speculative decoding inference. Components marked with the snowflake remain frozen during training to preserve their parameters, while components with the flame are trainable. This architecture enables efficient knowledge transfer from the target vision-language model to the smaller draft model while maintaining alignment in their token distributions.
• Figure 3: Mean accepted lengths when drafting for Qwen2.5-VL 7B Instruct at temperature $T = 0$ with speculation length $\gamma = 5$. The baseline uses Qwen2.5-1.5B as a text-only drafter (image tokens removed). MASSV achieves a substantial improvement in token acceptance across all tasks, increasing overall mean accepted length from 2.46 to 3.20 (+30.1%).
• Figure 4: Histogram of total variation distances (TVD), comparing the Qwen2.5-1.5B drafters trained with (purple) and without (orange) self-distilled visional instruction (SDViT) against the Qwen2.5-VL 7B target model on our multimodal SD benchmark. MASSV yields a highly skewed distribution concentrated at low TVD values, indicating tighter alignment with the target’s token distribution. In contrast, MASSV$_{\text{w/o SDViT}}$ produces a broader, heavier-tailed distribution, reflecting reduced alignment. The left-skewed shape of the MASSV distribution quantitatively suggests that SDViT narrows the distributional gap between draft and target.
• Figure 5: Training loss curves obtained during the two-phase MASSV training process when adapting Qwen2.5-1.5B Instruct into a VLM drafter for Qwen2.5-VL 7B Instruct. (a) shows the cross-entropy loss during multimodal projector pretraining, which rapidly decreases from $\sim$8.0 to $\sim$2.5 within 2000 steps, indicating efficient adaptation of the trainable projector. (b) displays the loss trajectory during fine-tuning with self-generated target VLM responses, with stable convergence around 1.1 across 5000 training steps, demonstrating successful token distribution alignment between the draft and target models.