Stateful Token Reduction for Long-Video Hybrid VLMs

TL;DR

This work proposes a low-to-high progressive reduction schedule and a unified language-aware scoring mechanism for both attention and Mamba blocks (using an implicit-attention proxy for Mamba), enabling all-layer token reduction in hybrids.

Abstract

Token reduction is an effective way to accelerate long-video vision-language models (VLMs), but most existing methods are designed for dense Transformers and do not directly account for hybrid architectures that interleave attention with linear-time state-space blocks (e.g., Mamba). We study query-conditioned token reduction for hybrid video VLMs and analyze reduction behavior through two properties: layerwise sparsity (how many tokens capture query-relevant information) and importance stability (whether token-importance rankings persist across depth). Although token importance is sparse within each layer, the set of important tokens changes across layers, so aggressive early pruning is unreliable. Motivated by this, we propose a low-to-high progressive reduction schedule and a unified language-aware scoring mechanism for both attention and Mamba blocks (using an implicit-attention proxy for Mamba), enabling all-layer token reduction in hybrids. Under an aggressive compression setting (retaining 25% of visual tokens), our approach delivers substantial prefilling speedups (3.8--4.2x) with near-baseline accuracy at test time, and light finetuning under reduction further improves performance on long-context video benchmarks.

Figures (6)

• Figure 1: Layer-Wise Attention Density and Cross-Layer Stability of Token-importance Rankings. Top: Kendall's $\tau$ correlation kendall1938new of token-importance rankings across layers for the Transformer model (a) and the Mamba--Transformer hybrid model (b). We consider values around 0.5 and below to indicate low cross-layer rank consistency. Bottom: Layer-wise density (%) of token-importance scores for the Transformer (c) and the hybrid model (d), where each dot corresponds to an individual attention head (blue) or a Mamba group (green). Lower density indicates higher sparsity.
• Figure 2: Hybrid Token Reduction Patterns used in our experiments. Layer types (Mamba/MLP/Attention) and reduction locations for baseline, attention-only, Mamba-only, and hybrid schedules (All attn+1M/All attn+2M).
• Figure 3: Hybrid reduction schedules used in our experiments. We visualize how the token retention ratio changes with depth for Nemotron-Nano-V2 VL 12B. Curves compare (1) no reduction, (2) step decay at attention layers, (3) step decay at attention + intervening Mamba layers, and (4) progressive low-to-high reduction applied throughout the network. Vertical dotted lines indicate attention layers locations.
• Figure 4: Latency Analysis for Nemotron-Nano-V2 VL 12B (hybrid).(left) LLM-stage latency vs. video frames on a single A100, with/without all-layer token reduction. (right) Component-wise latency (vision encoder / projector / LLM) for baseline vs. all-layer reduction on a 256-frame input.
• Figure 5: Latency Analysis for Qwen3-VL 8B (Transformer).(left) LLM-stage latency vs. frames on a single A100, with/without token reduction. (right) Component-wise latency (vision encoder / projector / LLM) on a 256-frame VideoMME input.