SparseVILA: Decoupling Visual Sparsity for Efficient VLM Inference

TL;DR

SparseVILA tackles the latency overhead in Vision-Language Models caused by dense visual token processing. It introduces a decoupled sparsity framework that prunes visually redundant tokens during prefill in a query-agnostic manner and performs query-aware retrieval from a preserved KV cache during decoding. This separation aligns sparsity with the distinct costs of context construction and autoregressive generation, enabling large end-to-end speedups while maintaining multi-turn fidelity across image, video, and reasoning tasks. The approach relies on efficient token-salience estimation, fast fused kernels, and careful RoPE handling, achieving up to 4.0x prefilling, 2.5x decoding, and 2.6x end-to-end speedups with robust accuracy gains, making it a practical, architecture-agnostic solution for scalable multimodal inference.

Abstract

Vision Language Models (VLMs) have rapidly advanced in integrating visual and textual reasoning, powering applications across high-resolution image understanding, long-video analysis, and multi-turn conversation. However, their scalability remains limited by the growing number of visual tokens that dominate inference latency. We present SparseVILA, a new paradigm for efficient VLM inference that decouples visual sparsity across the prefilling and decoding stages. SparseVILA distributes sparsity across stages by pruning redundant visual tokens during prefill and retrieving only query-relevant tokens during decoding. This decoupled design matches leading prefill pruning methods while preserving multi-turn fidelity by retaining most of the visual cache so that query-aware tokens can be retrieved at each conversation round. Built on an AWQ-optimized inference pipeline, SparseVILA achieves up to 4.0 times faster prefilling, 2.5 times faster decoding, and an overall 2.6 times end-to-end speedup on long-context video tasks -- while improving accuracy on document-understanding and reasoning tasks. By decoupling query-agnostic pruning and query-aware retrieval, SparseVILA establishes a new direction for efficient multimodal inference, offering a training-free, architecture-agnostic framework for accelerating large VLMs without sacrificing capability.

Figures (8)

• Figure 2: Multi-Round Query-Aware Comparisons with LLaVA-1.5 liu2024llava on the POPE li2023pope dataset. Without re-prefilling the context, the query-aware oracle degrades heavily, indicating the inability of query-aware pruning to scale effectively in a multi-turn conversation.
• Figure 3: Latency distribution over the prefilling and decoding stages across image, video, and reasoning workloads.
• Figure 4: Overview of SparseVILA’s decoupled sparsity framework. In the prefill stage, query-agnostic pruning removes redundant visual tokens based on salience scores from the visual encoder, yielding a compact representation shared across conversation turns. During decoding, query-aware retrieval selects only the most relevant visual tokens from the KV cache for attention computation, accelerating generation while maintaining multi-turn fidelity.
• Figure 5: Salience Metric Kernels. Latency comparison between the naïve and custom Triton implementations across two settings: (a) query-agnostic salience computation for the SigLIP and QwenVL vision encoders, and (b) query-aware retrieval salience for the Llama2 and Qwen2 decoder backbones. Our custom kernels consistently accelerate both query-agnostic and retrieval salience computations, achieving up to 10$\times$ and 1.8$\times$ speedups, respectively.
• Figure 6: Visual Retrieval Results. SparseVLM and FastV degrade and fail beyond 32 frames (8K context), while SparseVILA maintains perfect retrieval up to 200 frames, demonstrating superior long-context scalability.