CATP: Contextually Adaptive Token Pruning for Efficient and Enhanced Multimodal In-Context Learning

TL;DR

The paper tackles the severe image token redundancy in multimodal in-context learning by introducing Contextually Adaptive Token Pruning (CATP), a training‑free two‑stage token pruning method. Stage 1 selects tokens after the projector by jointly optimizing diversity and alignment with paired text, while Stage 2 uses progressive adaptation in shallow decoder layers to distill context and prune query tokens, aligning with cross‑modal interactions. Across eight benchmarks and four LVLMs, CATP consistently outperforms baselines and reduces latency, establishing a practical, efficient approach for interleaved image–text ICL. This work highlights the importance of modeling cross‑modal interactions in token pruning and provides a foundation for broader deployment in LVLMs with long context.

Abstract

Modern large vision-language models (LVLMs) convert each input image into a large set of tokens that far outnumber the text tokens. Although this improves visual perception, it also introduces severe image token redundancy. Because image tokens contain sparse information, many contribute little to reasoning but greatly increase inference cost. Recent image token pruning methods address this issue by identifying important tokens and removing the rest. These methods improve efficiency with only small performance drops. However, most of them focus on single-image tasks and overlook multimodal in-context learning (ICL), where redundancy is higher and efficiency is more important. Redundant tokens weaken the advantage of multimodal ICL for rapid domain adaptation and lead to unstable performance. When existing pruning methods are applied in this setting, they cause large accuracy drops, which exposes a clear gap and the need for new approaches. To address this, we propose Contextually Adaptive Token Pruning (CATP), a training-free pruning method designed for multimodal ICL. CATP uses two stages of progressive pruning that fully reflect the complex cross-modal interactions in the input sequence. After removing 77.8% of the image tokens, CATP achieves an average performance gain of 0.6% over the vanilla model on four LVLMs and eight benchmarks, clearly outperforming all baselines. At the same time, it improves efficiency by reducing inference latency by an average of 10.78%. CATP strengthens the practical value of multimodal ICL and lays the foundation for future progress in interleaved image-text settings.

Figures (12)

• Figure 1: (a) Example of a 3-shot in-context sequence. (b-d) show the effects of three image token pruning methods on this sequence. Attention-based methods tend to keep tokens at the bottom of the image due to attention sinks in interleaved inputs. Diversity-based methods lack semantic guidance from the entire context. In contrast, our proposed method CATP accounts for the complex cross-modal interactions within the sequence, leading to superior results.
• Figure 2: Performance comparison of three token pruning methods under a 77.8% per‑image pruning ratio, shown for single‑image versus 4‑shot ICL across six benchmarks.
• Figure 3: Average relative performance of pruning methods in 4-shot ICL: (a) showing diversity‑based pruning at 50%-90% ratios; (b-c) showing three different attention‑based pruning methods applied to various layers at 80% and 90% ratios.
• Figure 4: An overview pipeline of CATP. The version with more details is provided in Appendix 1.
• Figure 5: Performance of CATP on four LVLMs across diverse: (a) shot counts and (b) ICD selection strategies.