Don't Just Chase "Highlighted Tokens" in MLLMs: Revisiting Visual Holistic Context Retention
Xin Zou, Di Lu, Yizhou Wang, Yibo Yan, Yuanhuiyi Lyu, Xu Zheng, Linfeng Zhang, Xuming Hu
TL;DR
The paper addresses the high computational cost of multimodal large language models by criticizing attention-based token pruning that favors locally salient tokens. It introduces HoloV, a crop-based, holistic-context token pruning framework that adaptively allocates pruning budgets across image crops by combining intra-crop diversity and CLS attention, and optionally refetches pruned context via a fast FFN-based memory mechanism. Empirical results across diverse benchmarks and architectures show HoloV achieves strong efficiency-accuracy trade-offs, preserving near-original performance even at very high pruning ratios (e.g., 88.9% token reduction with ~4% drop) and generalizing to different backbones like LLaVA and Qwen. The approach promises practical deployment improvements for resource-constrained settings while maintaining robust visual reasoning capabilities, with analyses on complexity, theory, and visualization guiding its design and future extensions.
Abstract
Despite their powerful capabilities, Multimodal Large Language Models (MLLMs) suffer from considerable computational overhead due to their reliance on massive visual tokens. Recent studies have explored token pruning to alleviate this problem, which typically uses text-vision cross-attention or [\texttt{CLS}] attention to assess and discard redundant visual tokens. In this work, we identify a critical limitation of such attention-first pruning approaches, i.e., they tend to preserve semantically similar tokens, resulting in pronounced performance drops under high pruning ratios. To this end, we propose {HoloV}, a simple yet effective, plug-and-play visual token pruning framework for efficient inference. Distinct from previous attention-first schemes, HoloV rethinks token retention from a holistic perspective. By adaptively distributing the pruning budget across different spatial crops, HoloV ensures that the retained tokens capture the global visual context rather than isolated salient features. This strategy minimizes representational collapse and maintains task-relevant information even under aggressive pruning. Experimental results demonstrate that our HoloV achieves superior performance across various tasks, MLLM architectures, and pruning ratios compared to SOTA methods. For instance, LLaVA1.5 equipped with HoloV preserves 95.8\% of the original performance after pruning 88.9\% of visual tokens, achieving superior efficiency-accuracy trade-offs.