VisionTrim: Unified Vision Token Compression for Training-Free MLLM Acceleration

TL;DR

VisionTrim addresses the high computational cost of multimodal LLMs driven by abundant visual tokens. It introduces two plug-and-play modules, DVTS and TGVC, to prune and complement tokens across the entire MLLM pipeline in a training-free manner, using global semantic signals, local spatial affinity, and text-guided clustering with cross-modal alignment. The approach yields substantial efficiency gains with minimal performance loss across image and video benchmarks, including high-resolution and video settings, and generalizes to multiple MLLMs. This work enables practical deployment of efficient MLLMs by dramatically reducing token counts and inference costs while preserving cross-modal understanding. The authors also release code for reproducibility and provide extensive ablations and efficiency analyses to support their claims.

Abstract

Multimodal large language models (MLLMs) suffer from high computational costs due to excessive visual tokens, particularly in high-resolution and video-based scenarios. Existing token reduction methods typically focus on isolated pipeline components and often neglect textual alignment, leading to performance degradation. In this paper, we propose VisionTrim, a unified framework for training-free MLLM acceleration, integrating two effective plug-and-play modules: 1) the Dominant Vision Token Selection (DVTS) module, which preserves essential visual tokens via a global-local view, and 2) the Text-Guided Vision Complement (TGVC) module, which facilitates context-aware token merging guided by textual cues. Extensive experiments across diverse image and video multimodal benchmarks demonstrate the performance superiority of our VisionTrim, advancing practical MLLM deployment in real-world applications. The code is available at: https://github.com/hanxunyu/VisionTrim.

Figures (16)

• Figure 1: Comparison of previous methods with VisionTrim. (a) Previous methods focus solely on a specific part of the MLLM framework, typically the vision encoding or LLM decoding stages. (b) In contrast, VisionTrim optimizes the entire MLLM pipeline by introducing two plug-and-play modules, Dominant Vision Token Selection (DVTS) and Text-Guided Vision Complement (TGVC), to effectively reduce visual tokens in both the vision encoding and LLM decoding phases.
• Figure 2: Performance of VisionTrim. (a) Comparison across 10 benchmarks using the standard LLaVA-1.5-7B llava1.5, with an 88.9% reduction in visual tokens. (b) & (c) Performance vs. efficiency of various methods with a range of visual tokens, in both training-free and fine-tuning scenarios, respectively. The fine-tuning VisionTrim (Ours${\ddagger}$) demonstrates superior performance over previous image- and video-based MLLMs.
• Figure 3: (a) Overview of VisionTrim featuring the detailed DVTS module, and (b) the structure of the TGVC module. Both DVTS and TGVC modules can be generally utilized in both the vision encoding stage and the LLM decoding stage.
• Figure 4: Comparison of attention maps during LLM forward processing, with and without our proposed VisionTrim.
• Figure 5: Visualization of retained visual patches with and without the dual-attention mechanism in the DVTS module. Black-masked areas indicate discarded visual tokens.