TrimTokenator-LC: Towards Adaptive Visual Token Pruning for Large Multimodal Models with Long Contexts

TL;DR

This work addresses the cost of long-context visual inputs in large multimodal models by introducing TrimTokenator-LC, an adaptive two-stage visual token pruning framework. It decomposes redundancy into intra-image diversity and inter-image variation to allocate budgets dynamically, performing intra-image greedy selection followed by inter-image global pruning with Pareto optimization constrained by text alignment. The approach yields a compact, diverse, and text-aligned visual token set that preserves reasoning capabilities across long-context, multi-image scenarios, while achieving substantial speedups and memory savings in extensive experiments across multiple models and tasks. The findings demonstrate robust performance gains and practical efficiency improvements, motivating deployment of long-context LMMs under tighter compute budgets. The method also offers a clear hierarchy of design choices and ablation evidence, supporting the necessity of both stages and the Pareto-based cross-image filtering for effective pruning.

Abstract

Large Multimodal Models (LMMs) have proven effective on various tasks. They typically encode visual inputs into Original Model sequences of tokens, which are then concatenated with textual tokens and jointly processed by the language model. However, the growing number of visual tokens greatly increases inference cost. Visual token pruning has emerged as a promising solution. However, existing methods often overlook scenarios involving long context inputs with multiple images. In this paper, we analyze the challenges of visual token pruning in long context, multi-image settings and introduce an adaptive pruning method tailored for such scenarios. We decompose redundancy into intra-image and inter-image components and quantify them through intra-image diversity and inter-image variation, which jointly guide dynamic budget allocation. Our approach consists of two stages. The intra-image stage allocates each image a content-aware token budget and greedily selects its most representative tokens. The inter-image stage performs global diversity filtering to form a candidate pool and then applies a Pareto selection procedure that balances diversity with text alignment. Extensive experiments show that our approach maintains strong performance in long context settings while significantly cutting down the number of visual tokens.

Figures (4)

• Figure 1: Comprehensive overview of the pruning method. We first perform an Intra-image Stage, where the visual token budget is adaptively allocated based on the diversity of each image. Within each image, a greedy representative selection strategy is applied to retain the most representative local visual tokens. Next, in the Inter-image Stage, greedy representative filtering is conducted over the candidate tokens collected from all images. Finally, by jointly considering token diversity and semantic alignment with the text, pareto selection is used to preserve tokens that are both informative across images and highly aligned with the text. Figure \ref{['f_2']} provides a brief description.
• Figure 2: Brief overview of the visual token pruning method. Figure \ref{['f_1']} provides a detailed description.
• Figure 3: Performance on the ALFRED across different token retention ratios with LLaVA-1.5-7B.
• Figure 4: Case study of our pruning strategy on several LLaVA-1.5-7B samples. GT denotes Ground Truth.