SwiftVLM: Efficient Vision-Language Model Inference via Cross-Layer Token Bypass

TL;DR

SwiftVLM addresses the inefficiency of vision-language models by revealing that visual token importance is non-monotonic across layers and that early pruning can erase task-critical details. It introduces a bypass pruning paradigm and a training-free method that selects pruning layers via dynamic programming and preserves unselected tokens for re-evaluation at deeper layers, enabling independent pruning decisions per stage. Across nine benchmarks and multiple VLMs, SwiftVLM delivers superior accuracy-efficiency trade-offs, with notable gains on localization tasks and faithful visual token selection. The approach provides a practical, scalable path to faster VLM inference and suggests cross-layer token strategies as a promising direction for future pruning research.

Abstract

Visual token pruning is a promising approach for reducing the computational cost of vision-language models (VLMs), and existing methods often rely on early pruning decisions to improve efficiency. While effective on coarse-grained reasoning tasks, they suffer from significant performance degradation on tasks requiring fine-grained visual details. Through layer-wise analysis, we reveal substantial discrepancies in visual token importance across layers, showing that tokens deemed unimportant at shallow layers can later become highly relevant for text-conditioned reasoning. To avoid irreversible critical information loss caused by premature pruning, we introduce a new pruning paradigm, termed bypass, which preserves unselected visual tokens and forwards them to subsequent pruning stages for re-evaluation. Building on this paradigm, we propose SwiftVLM, a simple and training-free method that performs pruning at model-specific layers with strong visual token selection capability, while enabling independent pruning decisions across layers. Experiments across multiple VLMs and benchmarks demonstrate that SwiftVLM consistently outperforms existing pruning strategies, achieving superior accuracy-efficiency trade-offs and more faithful visual token selection behavior.

Figures (8)

• Figure 1: Comparison of visual token pruning strategies in VLMs. (a)–(b) Existing approaches suffer from irreversible loss of critical visual information once tokens are merged or dropped in shallow layers. (c) We propose Bypass, a pruning strategy that restores previously merged tokens via token alignment. Bypass provides critical visual tokens with an opportunity to be reconsidered at deeper layers with stronger token selection capability.
• Figure 2: Layer-wise variation in visual token ranking. For a representative TextVQA example, we report the overlap ratio between the bottom-ranked 50% of visual tokens selected at layers 1–9 and the top-ranked 10% selected at layers 10–20 of LLaVA.
• Figure 3: Comparison of results from different pruning methods. FastV applies aggressive early-layer pruning, whereas PDrop adopts progressive pruning. Both drop the visual token containing “NASRI”, leading to incorrect answers. SwiftVLM preserves the query-relevant token at the final stage and answers correctly.
• Figure 4: Non-monotonic layer-wise capability for visual token selection. Across tasks and datasets, we record the layer-wise top 20% visual tokens of the vanilla model and re-evaluate it by retaining all tokens in layers 1–2 and only the layer-specific top 20% from layer 3 onward. Performance is reported relative to the vanilla baseline.
• Figure 5: SwiftVLM architecture overview. (a) After layer $x$, unselected visual tokens are grouped for bypassing, with the resulting merged tokens participating in subsequent computation. (b) Before layer $y$, token alignment is applied to restore grouped tokens, enabling re-evaluation of visual tokens at layers with stronger token selection capability.