Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MMTok: Multimodal Coverage Maximization for Efficient Inference of VLMs

Sixun Dong, Juhua Hu, Mian Zhang, Ming Yin, Yanjie Fu, Qi Qian

TL;DR

This work proposes to leverage both vision and text tokens to select informative vision tokens by the coverage criterion, and achieves a 1.87x speedup while maintaining 98.7% of the original performance on LLaVA-NeXT-13B and highlights the effectiveness of coverage in token selection.

Abstract

Vision-Language Models (VLMs) demonstrate impressive performance in understanding visual content with language instruction by converting visual inputs to vision tokens. However, redundancy in vision tokens results in the degraded inference efficiency of VLMs. While many algorithms have been proposed to reduce the number of vision tokens, most of them apply only unimodal information (i.e., vision/text) for pruning and ignore the inherent multimodal property of vision-language tasks. Moreover, it lacks a generic criterion that can be applied to different modalities. To mitigate this limitation, in this work, we propose to leverage both vision and text tokens to select informative vision tokens by the coverage criterion. We first formulate the subset selection problem as a maximum coverage problem. Afterwards, a subset of vision tokens is optimized to cover the text tokens and the original set of vision tokens, simultaneously. The proposed method MMTok is extensively evaluated on benchmark datasets with different VLMs. The comparison illustrates that vision and text information are complementary, and combining multimodal information can surpass the unimodal baseline with a clear margin. Moreover, under the maximum coverage criterion on the POPE dataset, our method achieves a 1.87x speedup while maintaining 98.7% of the original performance on LLaVA-NeXT-13B. Finally, with only four vision tokens, 87.7% of the original performance is still preserved on LLaVA-1.5-7B. These results highlight the effectiveness of coverage in token selection. The code is available at https://github.com/Ironieser/mmtok

MMTok: Multimodal Coverage Maximization for Efficient Inference of VLMs

TL;DR

This work proposes to leverage both vision and text tokens to select informative vision tokens by the coverage criterion, and achieves a 1.87x speedup while maintaining 98.7% of the original performance on LLaVA-NeXT-13B and highlights the effectiveness of coverage in token selection.

Abstract

Vision-Language Models (VLMs) demonstrate impressive performance in understanding visual content with language instruction by converting visual inputs to vision tokens. However, redundancy in vision tokens results in the degraded inference efficiency of VLMs. While many algorithms have been proposed to reduce the number of vision tokens, most of them apply only unimodal information (i.e., vision/text) for pruning and ignore the inherent multimodal property of vision-language tasks. Moreover, it lacks a generic criterion that can be applied to different modalities. To mitigate this limitation, in this work, we propose to leverage both vision and text tokens to select informative vision tokens by the coverage criterion. We first formulate the subset selection problem as a maximum coverage problem. Afterwards, a subset of vision tokens is optimized to cover the text tokens and the original set of vision tokens, simultaneously. The proposed method MMTok is extensively evaluated on benchmark datasets with different VLMs. The comparison illustrates that vision and text information are complementary, and combining multimodal information can surpass the unimodal baseline with a clear margin. Moreover, under the maximum coverage criterion on the POPE dataset, our method achieves a 1.87x speedup while maintaining 98.7% of the original performance on LLaVA-NeXT-13B. Finally, with only four vision tokens, 87.7% of the original performance is still preserved on LLaVA-1.5-7B. These results highlight the effectiveness of coverage in token selection. The code is available at https://github.com/Ironieser/mmtok

Paper Structure

This paper contains 26 sections, 3 theorems, 10 equations, 5 figures, 22 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. The Proposed Method
  4. Vision Token Selection by Coverage Maximization
  5. Maximum Text-Vision Coverage
  6. Maximum Vision-Vision Coverage
  7. Maximum Multimodal Coverage
  8. Experiments
  9. Performance Comparison on Diverse Tasks
  10. Comparison on High IC Tasks with Limited Vision Tokens
  11. Ablation Study
  12. Conclusion
  13. Ethics statement
  14. Reproducibility statement
  15. Acknowledgments
  16. ...and 11 more sections

Key Result

Proposition 1

LeskovecKGFVG07 For all subsets $\mathcal{A}\subseteq\mathcal{B}\subseteq\mathcal{N}$ and $s\in\mathcal{N}\setminus\mathcal{B}$,

Figures (5)

  • Figure 1: MMTok demonstrates better performance across multiple benchmarks.
  • Figure 2: Overview of MMTok framework. Our method optimizes two maximum coverage problems simultaneously to leverage text-vision and vision-vision similarity for vision token selections.
  • Figure 3: Multi-turn conversation by applying MMTok only with text from Q1.
  • Figure 4: Answer changes with different number of tokens. Hard questions need more vision tokens.
  • Figure 5: Visualization of Selected Tokens. Compared with the diversity-based method, DivPrune, our method selected token coverage the necessary token associate to language context.

Theorems & Definitions (4)

  • Proposition 1
  • Proposition 2
  • Corollary 1
  • proof