Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Adaptive-VoCo: Complexity-Aware Visual Token Compression for Vision-Language Models

Xiaoyang Guo, Keze Wang

TL;DR

Adaptive-VoCo addresses the rigid token budgets in vision-language models by introducing a complexity-aware Rate Predictor that dynamically allocates visual tokens per image. It leverages vision-encoder signals, including patch-token entropy and attention variance, and expands a <voco> placeholder into a variable number of tokens, guided by a dual loss balancing efficiency and fidelity. End-to-end training yields higher average retention (89.3%) across seven benchmarks compared with fixed-rate baselines, while maintaining favorable inference costs. This approach enables more efficient and robust cross-modal reasoning by tailoring representation capacity to input difficulty.

Abstract

In recent years, large-scale vision-language models (VLMs) have demonstrated remarkable performance on multimodal understanding and reasoning tasks. However, handling high-dimensional visual features often incurs substantial computational and memory costs. VoCo-LLaMA alleviates this issue by compressing visual patch tokens into a few VoCo tokens, reducing computational overhead while preserving strong cross-modal alignment. Nevertheless, such approaches typically adopt a fixed compression rate, limiting their ability to adapt to varying levels of visual complexity. To address this limitation, we propose Adaptive-VoCo, a framework that augments VoCo-LLaMA with a lightweight predictor for adaptive compression. This predictor dynamically selects an optimal compression rate by quantifying an image's visual complexity using statistical cues from the vision encoder, such as patch token entropy and attention map variance. Furthermore, we introduce a joint loss function that integrates rate regularization with complexity alignment. This enables the model to balance inference efficiency with representational capacity, particularly in challenging scenarios. Experimental results show that our method consistently outperforms fixed-rate baselines across multiple multimodal tasks, highlighting the potential of adaptive visual compression for creating more efficient and robust VLMs.

Adaptive-VoCo: Complexity-Aware Visual Token Compression for Vision-Language Models

TL;DR

Adaptive-VoCo addresses the rigid token budgets in vision-language models by introducing a complexity-aware Rate Predictor that dynamically allocates visual tokens per image. It leverages vision-encoder signals, including patch-token entropy and attention variance, and expands a <voco> placeholder into a variable number of tokens, guided by a dual loss balancing efficiency and fidelity. End-to-end training yields higher average retention (89.3%) across seven benchmarks compared with fixed-rate baselines, while maintaining favorable inference costs. This approach enables more efficient and robust cross-modal reasoning by tailoring representation capacity to input difficulty.

Abstract

In recent years, large-scale vision-language models (VLMs) have demonstrated remarkable performance on multimodal understanding and reasoning tasks. However, handling high-dimensional visual features often incurs substantial computational and memory costs. VoCo-LLaMA alleviates this issue by compressing visual patch tokens into a few VoCo tokens, reducing computational overhead while preserving strong cross-modal alignment. Nevertheless, such approaches typically adopt a fixed compression rate, limiting their ability to adapt to varying levels of visual complexity. To address this limitation, we propose Adaptive-VoCo, a framework that augments VoCo-LLaMA with a lightweight predictor for adaptive compression. This predictor dynamically selects an optimal compression rate by quantifying an image's visual complexity using statistical cues from the vision encoder, such as patch token entropy and attention map variance. Furthermore, we introduce a joint loss function that integrates rate regularization with complexity alignment. This enables the model to balance inference efficiency with representational capacity, particularly in challenging scenarios. Experimental results show that our method consistently outperforms fixed-rate baselines across multiple multimodal tasks, highlighting the potential of adaptive visual compression for creating more efficient and robust VLMs.

Paper Structure

This paper contains 13 sections, 8 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Evolution of Fixed-Rate Compression
  4. Limitation of Static Token Allocation
  5. Method
  6. Complexity-Aware Rate Prediction
  7. Adaptive Token Allocation
  8. Loss Formulation for Rate Predictor Training
  9. Experiments
  10. Experimental Setup
  11. Main Results
  12. Compression–Performance–Efficiency Trade-off
  13. Conclusion

Figures (2)

  • Figure 1: Overview of the proposed Adaptive-VoCo framework. The input image is processed by a vision encoder to extract the class token (CLS), patch-level embeddings, and the attention map. These features are fed into the Rate Predictor to estimate the appropriate number of compressed visual tokens $K_{\text{voco}}$. The predicted token number is then used to expand the placeholder token into $K_{\text{voco}}$ VoCo tokens. The Rate Predictor is trained with the Rate Loss ($L_{\text{rate}}$) and the Complexity Loss ($L_{\text{comp}}$) to ensure efficient and complexity-aware token allocation.
  • Figure 2: Comparison of Adaptive-VoCo with fixed-rate VoCo-LLaMA models.