A Survey of Token Compression for Efficient Multimodal Large Language Models

TL;DR

The paper presents the first systematic survey of token compression for multimodal large language models, targeting long-context efficiency across image, video, and audio modalities. It defines a formal taxonomy by modality (image/video/audio) and mechanism (transformation, similarity, attention, query) and surveys representative techniques within each category, including pixel unshuffle, token merging, attention-based pruning, and query-guided distillation. The analysis highlights current challenges such as performance degradation at high compression, deployment hurdles with optimized kernels, and evaluation gaps, while identifying future directions like joint multimodal compression and architecture innovations. The work aims to guide researchers and developers toward efficient, scalable MLLMs that can handle real-world long-context tasks with improved modality alignment and generalization.

Abstract

Multimodal large language models (MLLMs) have made remarkable strides, largely driven by their ability to process increasingly long and complex contexts, such as high-resolution images, extended video sequences, and lengthy audio input. While this ability significantly enhances MLLM capabilities, it introduces substantial computational challenges, primarily due to the quadratic complexity of self-attention mechanisms with numerous input tokens. To mitigate these bottlenecks, token compression has emerged as an auspicious and critical approach, efficiently reducing the number of tokens during both training and inference. In this paper, we present the first systematic survey and synthesis of the burgeoning field of multimodal long context token compression. Recognizing that effective compression strategies are deeply tied to the unique characteristics and redundancies of each modality, we categorize existing approaches by their primary data focus, enabling researchers to quickly access and learn methods tailored to their specific area of interest: (1) image-centric compression, which addresses spatial redundancy in visual data; (2) video-centric compression, which tackles spatio-temporal redundancy in dynamic sequences; and (3) audio-centric compression, which handles temporal and spectral redundancy in acoustic signals. Beyond this modality-driven categorization, we further dissect methods based on their underlying mechanisms, including transformation-based, similarity-based, attention-based, and query-based approaches. By providing a comprehensive and structured overview, this survey aims to consolidate current progress, identify key challenges, and inspire future research directions in this rapidly evolving domain.

Figures (4)

• Figure 1: Left: Image, video, and audio data types can scale in their representation dimensions, leading to a corresponding increase in the number of tokens. Right: Top-performing MLLMs cannot address real-world demands, as the number of tokens for multimodal input, especially video, vastly exceeds that of text, and most visual tokens are redundant. Therefore, token compression is crucial to address this limitation.
• Figure 2: Representative Architecture of MLLMs. Within MLLM reasoning processes, token sequences comprise concatenated system tokens, multimodal tokens, and text tokens. Multimodal tokens usually constitute the majority of the sequence tokens.
• Figure 3: Taxonomy of Multimodal Token Compression. Our classification organizes existing methods by their dominant data modality, accounting for inherent differences in redundancy across modalities. This is further refined by a dissection of their underlying mechanisms, enabling researchers to quickly pinpoint methods tailored to specific research domains.
• Figure 4: Trade-off between Retained Ratio and Performance across Modalities.Left: We visualize changes in token retention and model performance on the VQA$^{2}$goyal2017making for image LLMs using each method’s reported setup with LLaVA-1.5-7Bliu2023visual. Right: For video LLMs, we plot the video-token retention ratio and the corresponding performance deltas on the VideoMME benchmark fu2025video, following each method’s reported configuration with LLaVA-OV-7Bli2024llava. As different methods target distinct compression regimes, we primarily report results at the compression rates specified in their original papers.