ConsensusDrop: Fusing Visual and Cross-Modal Saliency for Efficient Vision Language Models

TL;DR

ConsensusDrop tackles the high inference cost of vision-language models by pruning visual tokens before the language model. It fuses vision-encoder saliency with query-aware cross-attention via a training-free pipeline (SCAP, Fuser, EGTM) to obtain a robust consensus ranking and compact visual representations. The method delivers state-of-the-art accuracy–efficiency trade-offs across LLaVA-1.5, LLaVA-NeXT, Video-LLaVA, and other open-source VLMs, preserving near-baseline performance even at aggressive token reductions while substantially reducing TTFT and KV-cache footprint. Its training-free design enables easy deployment to higher-resolution and video inputs, with open-source release planned to support broad adoption and reproducibility.

Abstract

Vision-Language Models (VLMs) are expensive because the LLM processes hundreds of largely redundant visual tokens. Existing token reduction methods typically exploit \textit{either} vision-encoder saliency (broad but query-agnostic) \textit{or} LLM cross-attention (query-aware but sparse and costly). We show that neither signal alone is sufficient: fusing them consistently improves performance compared to unimodal visual token selection (ranking). However, making such fusion practical is non-trivial: cross-modal saliency is usually only available \emph{inside} the LLM (too late for efficient pre-LLM pruning), and the two signals are inherently asymmetric, so naive fusion underutilizes their complementary strengths. We propose \textbf{ConsensusDrop}, a training-free framework that derives a \emph{consensus} ranking by reconciling vision encoder saliency with query-aware cross-attention, retaining the most informative tokens while compressing the remainder via encoder-guided token merging. Across LLaVA-1.5/NeXT, Video-LLaVA, and other open-source VLMs, ConsensusDrop consistently outperforms prior pruning methods under identical token budgets and delivers a stronger accuracy-efficiency Pareto frontier -- preserving near-baseline accuracy even at aggressive token reductions while reducing TTFT and KV cache footprint. Our code will be open-sourced.

Figures (30)

• Figure 1: Motivation: Complementary failure modes of visual token sparsification. Given a single image and two questions requiring distinct visual evidence (global structure vs. localized fine-grained text), different token selection criteria exhibit systematic limitations. Vision-only saliency preserves broad spatial coverage and visually salient regions, but may discard small, text-critical regions. Text--vision saliency captures query-aligned local details, but is often sparse and biased towards late-index patches. ConsensusDrop (ours) retains complementary evidence by fusing visual and cross-modal saliency signals, preserving information required across diverse questions.
• Figure 2: Multimodal recovery consistently improves accuracy across benchmarks. Using LLaVA-1.5-7B, we compare token selection based on (i) cross-modal scores, (ii) vision scores, and (iii) our recovery mechanism that fuses the two. Across POPE, MME, MMBench, MMBench-CN, TextVQA, and GQA, recovery achieves higher accuracy at all retention ratios (0.25 and 0.50) and for both student configurations (visual and cross-modal). These validate that neither unimodal signal is sufficient, and that combining them provides a more reliable estimate of token importance.
• Figure 3: Three-way relationship between disagreement, correction rate, and accuracy improvement. Bubble size denotes disagreement between visual and cross-modal attention; color distinguishes retention ratio ($\rho=0.25$ vs. $\rho=0.5$). Higher disagreement consistently leads to higher correction, which yields greater accuracy gains. Low-retention (red) exhibit the strongest effect, confirming that aggressive pruning amplifies productive modal disagreements that consensus recovery can leverage.
• Figure 4: Correction-rate decay as the recovery rate $r$ increases for POPE and GQA. Correction rates are highest at small $r$, where nearly all recovered tokens correspond to true multimodal disagreements. As $r$ increases, the benefit of recovery saturates: additional tokens reintroduced are increasingly ones that both modalities already agreed upon. This reveals a strong quality--quantity tradeoff and shows that consensus-based recovery naturally prioritizes a small set of high-value corrections.
• Figure 5: Illustration of ConsensusDrop. Given an image--question pair, a frozen vision encoder and projector produce visual tokens, and SCAP provides query-aware cross-attention scores. A lightweight fuser combines vision and cross-modal saliency to select top-$K$ tokens, while EGTM merges the remaining tokens into compact representations, yielding a compressed visual sequence for the LLM.