AdaTok: Adaptive Token Compression with Object-Aware Representations for Efficient Multimodal LLMs

TL;DR

AdaTok presents an object-aware token compression method for multimodal LLMs that merges image representations at the object level rather than patch-level, aligning with human vision and reducing redundant tokens. By leveraging SAM for object masks and a lightweight projector, AdaTok produces a compact set of object tokens that preserve semantic content and reduce computation; it attains roughly 96% of a vanilla model's performance using about 10% of the tokens, across multiple benchmarks. The approach avoids changes to the core architecture, demonstrates strong token efficiency, and shows potential for edge deployment and reduced communication bandwidth, while noting limitations in OCR tasks and suggesting future improvements with text-recognition models. Overall, AdaTok offers a scalable, adaptable, and robust strategy for efficient multimodal reasoning with large language models.

Abstract

Multimodal Large Language Models (MLLMs) have demonstrated substantial value in unified text-image understanding and reasoning, primarily by converting images into sequences of patch-level tokens that align with their architectural paradigm. However, patch-level tokenization leads to a quadratic growth in image tokens, burdening MLLMs' understanding and reasoning with enormous computation and memory. Additionally, the traditional patch-wise scanning tokenization workflow misaligns with the human vision cognition system, further leading to hallucination and computational redundancy. To address this issue, we propose an object-level token merging strategy for Adaptive Token compression, revealing the consistency with human vision system. The experiments are conducted on multiple comprehensive benchmarks, which show that our approach averagely, utilizes only 10% tokens while achieving almost 96% of the vanilla model's performance. More extensive experimental results in comparison with relevant works demonstrate the superiority of our method in balancing compression ratio and performance. Our code will be available.

Figures (5)

• Figure 1: (a) The main idea of AdaTok. Our object-level image token compression approach inherently outperforms patch-level compression-based methods in terms of the balance on compression ratio and accuracy. (b) The x-axis is token numbers, y-axis is the total score on MME benchmark.
• Figure 2: The overview of our object-aware representation merging strategy. We implement our strategy based on LLaVA architecture.
• Figure 3: Token efficiency comparison on different benchmarks. Our proposed object-aware token merging strategy requires fewer tokens to maintain comparable performance, showcasing its inherent advantage over patch-level based compression strategies.
• Figure 4: Limitation of AdaTok on OCR task. It is challenging to extract the region of individual letter in the image. To address this issue, it is necessary to increase the number of sampling points to acquire more object masks.
• Figure 5: The visual question answering results under different tokens. For simple visual question answering tasks such as scene recognition and image captioning, AdaTok only requires the number of relevant key objects to complete the task. For fine-grained recognition tasks such as text recognition, it can also be accomplished by increasing the number of tokens in the image. We can adjust the number of tokens input to AdaTok to flexibly handle tasks of varying complexity. LLaVA-v1.5-7B is adopted here.