FastVLM: Self-Speculative Decoding for Fast Vision-Language Model Inference

TL;DR

FastVLM introduces a self-speculative decoding framework for vision–language models that uses a lightweight draft model augmented by an imitation network to mimic deeper representations, enabling verification by a full model. By decoupling draft and final-layer objectives and reusing a shared KV-cache, the approach maintains full-model accuracy while achieving substantial inference speedups of approximately $1.55$–$1.85\times$. The imitation network learns to recover deep-layer information via Cosine Similarity and knowledge distillation losses, with dynamic drafting parameters $d(t)$ and an empirical reward to select the draft depth $n$, validated across multiple VL tasks and decoders. This work offers a scalable path to faster VL inference on resource-constrained devices, with practical impact for real-time multimodal applications.

Abstract

Vision-language Models (VLMs) have made significant strides in visual understanding and query response generation, but often face challenges of high computational cost and inference latency due to autoregressive decoding. In this work, we introduce an imitation-learning-based Self-Speculative Decoding (SSD) framework, named FastVLM, to address these limitations. Our approach employs a lightweight draft model for token generation in an autoregressive manner, while a full model verifies these tokens non-autoregressively. Accepted tokens proceed seamlessly, while rejected tokens are corrected by the full model and used to guide the draft model's refinement. Through an imitation network, FastVLM enhances the draft model by integrating deeper level insights from the full model's architecture. Also, it maintains the performance integrity of the full model while training the draft model, achieving a balance between efficiency and accuracy. Our method speeds up the inference process by 1.55-1.85x as compared to the final layer with minimal loss in performance.

Figures (9)

• Figure 1: The effect on the final layer BLEU4 score due to combined training of the full model with the draft model.
• Figure 2: Inference from different model types: a) Draft without imitation; b) Draft with imitation; c) Full model; d) Self-Speculative Decoding.
• Figure 3: Overview of our method.Left: During training, the imitation network takes inputs from the $n$th decoder layer and learns to mimic deeper layer representations. Its output is passed to the LM Head, and training uses a combination of similarity loss, knowledge distillation, and cross-entropy loss. Right: At inference, the $n$th decoder layer and imitation network generate draft tokens, which are then verified by the full decoder.
• Figure 4: An instance of our method applied to LLaVA-1.5 on a long-context example.
• Figure 5: The token acceptance rates across different time-steps, i.e., at the $t$th token generation when $t-1$ tokens are used as a context.