On Speculative Decoding for Multimodal Large Language Models

TL;DR

This paper addresses the slow inference of multimodal large language models by applying speculative decoding (SPD) to the LLaVA-7B system. It demonstrates that a language-only draft model of 115M parameters can provide substantial speedups (up to 2.37× MBSU) without processing image tokens, while a compact image-adapter draft offers marginal gains in captioning. The work evaluates SPD across three tasks (LLaVA Instruct 150K, COCO captioning, and ScienceQA) and shows that text-only drafts often match or exceed image-informed drafts in many settings. The findings highlight SPD's potential to boost practical throughput for multimodal LLMs and open avenues for extending SPD to other models and modalities.

Abstract

Inference with Multimodal Large Language Models (MLLMs) is slow due to their large-language-model backbone which suffers from memory bandwidth bottleneck and generates tokens auto-regressively. In this paper, we explore the application of speculative decoding to enhance the inference efficiency of MLLMs, specifically the LLaVA 7B model. We show that a language-only model can serve as a good draft model for speculative decoding with LLaVA 7B, bypassing the need for image tokens and their associated processing components from the draft model. Our experiments across three different tasks show that speculative decoding can achieve a memory-bound speedup of up to 2.37$\times$ using a 115M parameter language model that we trained from scratch. Additionally, we introduce a compact LLaVA draft model incorporating an image adapter, which shows marginal performance gains in image captioning while maintaining comparable results in other tasks.

Figures (5)

• Figure 1: SPD with a MLLM as target having three components: vision encoder, image projector, and target language model, and the smaller language model as draft. The small draft model generates draft tokens autoregressively for block-size number of iterations followed by parallel evaluation by the target language model which also uses image features.
• Figure 2: MBSU, block efficiency and token rate (relative to auto-regressive generation) for SPD are depicted; We consider LLaVA-eval, COCO-Caption and SQA datasets for evaluation; For draft models, base-LLaMA, chat-LLaMA, ft-LLaVA-text, ft-LLaVA are considered, we consider three text-only draft models and a single text and image draft model; For draft length (DL) (or block size of SPD), we consider either 3 or 5.
• Figure 3: Example 1
• Figure 4: Example 2
• Figure : Example 1