VisionSelector: End-to-End Learnable Visual Token Compression for Efficient Multimodal LLMs

TL;DR

VisionSelector tackles the efficiency bottlenecks of Multimodal LLMs by reframing visual token compression as an end-to-end learnable decision problem. It introduces a lightweight Learnable Importance Scorer (LIS), a Differentiable Top-K (DTS) mechanism, and a Curriculum Annealing Strategy (CAS) to enable adaptive token pruning across budgets while preserving downstream performance. The approach achieves strong results across image- and video-language tasks, offering notable gains in speed and memory with minimal trainable parameters ($\approx$12.85M) and backbone compatibility. These capabilities suggest practical impact for deploying high-resolution or multi-image MLLMs in resource-constrained settings, with the potential to reduce computation without sacrificing accuracy.

Abstract

Multimodal Large Language Models (MLLMs) encounter significant computational and memory bottlenecks from the massive number of visual tokens generated by high-resolution images or multi-image inputs. Previous token compression techniques are often constrained by heuristic rules that risk discarding critical information. They may suffer from biases, such as attention sinks, that lead to sharp performance drops under aggressive compression ratios. To address these limitations, we reformulate token compression as a lightweight plug-and-play framework that reformulates token compression into an end-to-end learnable decision process. To be specific, we propose VisionSelector, a scorer module decoupled from the MLLM backbone that incorporates a differentiable Top-K mechanism and a curriculum annealing strategy to bridge the training-inference gap, enabling efficient and adaptive token selection various arbitrary compression rates. Remarkably lightweight with only 12.85M trainable parameters, VisionSelector demonstrates generalization across various compression rates and adaptively identifying critical tokens. This leads to superior performance across all compression budgets, evidenced by preserving 100% accuracy on MME with 30% retention budget, outperforming prior methods by 12.14% at 10% retention budget, and doubling prefill speed. Our code is available at https://github.com/JulietChoo/VisionSelector .

Figures (8)

• Figure 1: Performance and Efficiency of VisionSelector.(a) VisionSelector outperforms prior SOTA under a $10\%$ token budget on Qwen2.5-VL-7B, retaining $87.75\%$ performance on average. (b) Performance–speedup trade-off on DocVQA across 6 retention budgets. VisionSelector is optimal across all ratios, achieving a $2\times$ prefill speedup with only $5\%$ tokens. (c) DocVQA with a $20\%$ token budget: comparison of accuracy, GPU memory, and speedup. VisionSelector delivers a leading three‑way trade‑off.
• Figure 2: Overview of our VisionSelector. The framework introduces a lightweight, learnable importance scorer to evaluate token importance. During training, the Differentiable Top-K Selection produces a soft mask for gradient propagation, while a constraint loss integrated with a Curriculum Annealing Strategy progressively transforms it into a hard mask. At inference, the standard Top-K operation is applied for efficient token selection.
• Figure 3: Qualitative results of VisionSelector on TextVQA. The top row compares three methods for scoring token importance, where warmer colors (red) correspond to higher importance scores. The bottom row illustrates the tokens selected by each method and the corresponding model predictions at a $20\%$ compression budget.
• Figure 4: The impact of training and testing compression budgets on performance retention. The plot illustrates the model's performance robustness when trained under one compression budget (x-axis) and evaluated under another (each colored line). A key observation is that models generalize well to less strict compression budgets but suffer a significant performance degradation when subjected to stricter compression than they were trained on.
• Figure 5: Qualitative examples on the MME dataset under a $30\%$ retention budget: our method preserves informative regions and removes distracting “noise” tokens, yielding clearer attention and better answers than Qwen2.5‑VL‑7B across code reasoning, numerical calculation, and common-sense reasoning.