Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

STAR: Stage-Wise Attention-Guided Token Reduction for Efficient Large Vision-Language Models Inference

Yichen Guo, Hanze Li, Zonghao Zhang, Jinhao You, Kai Tang, Xiande Huang

TL;DR

STAR tackles the high cost of LVLM inference by introducing a stage-wise token reduction that is training-free and plug-and-play. It combines an early-stage visual self-attention pruning to remove low-level redundancy with a later-stage cross-modal attention pruning to discard tokens weakly related to the task, preserving crucial vision–language information. The approach yields substantial FLOPs reductions while maintaining or even improving performance across multiple LVLMs and benchmarks, including high-priority VQA datasets and larger-resolution models. This stage-aware strategy offers a practical path to deploying efficient LVLMs in real-world settings without retraining or extensive fine-tuning.

Abstract

Although large vision-language models (LVLMs) leverage rich visual token representations to achieve strong performance on multimodal tasks, these tokens also introduce significant computational overhead during inference. Existing training-free token pruning methods typically adopt a single-stage strategy, focusing either on visual self-attention or visual-textual cross-attention. However, such localized perspectives often overlook the broader information flow across the model, leading to substantial performance degradation, especially under high pruning ratios. In this work, we propose STAR (Stage-wise Attention-guided token Reduction), a training-free, plug-and-play framework that approaches token pruning from a global perspective. Instead of pruning at a single point, STAR performs attention-guided reduction in two complementary stages: an early-stage pruning based on visual self-attention to remove redundant low-level features, and a later-stage pruning guided by cross-modal attention to discard task-irrelevant tokens. This holistic approach allows STAR to significantly reduce computational cost while better preserving task-critical information. Extensive experiments across multiple LVLM architectures and benchmarks show that STAR achieves strong acceleration while maintaining comparable, and in some cases even improved performance.

STAR: Stage-Wise Attention-Guided Token Reduction for Efficient Large Vision-Language Models Inference

TL;DR

STAR tackles the high cost of LVLM inference by introducing a stage-wise token reduction that is training-free and plug-and-play. It combines an early-stage visual self-attention pruning to remove low-level redundancy with a later-stage cross-modal attention pruning to discard tokens weakly related to the task, preserving crucial vision–language information. The approach yields substantial FLOPs reductions while maintaining or even improving performance across multiple LVLMs and benchmarks, including high-priority VQA datasets and larger-resolution models. This stage-aware strategy offers a practical path to deploying efficient LVLMs in real-world settings without retraining or extensive fine-tuning.

Abstract

Although large vision-language models (LVLMs) leverage rich visual token representations to achieve strong performance on multimodal tasks, these tokens also introduce significant computational overhead during inference. Existing training-free token pruning methods typically adopt a single-stage strategy, focusing either on visual self-attention or visual-textual cross-attention. However, such localized perspectives often overlook the broader information flow across the model, leading to substantial performance degradation, especially under high pruning ratios. In this work, we propose STAR (Stage-wise Attention-guided token Reduction), a training-free, plug-and-play framework that approaches token pruning from a global perspective. Instead of pruning at a single point, STAR performs attention-guided reduction in two complementary stages: an early-stage pruning based on visual self-attention to remove redundant low-level features, and a later-stage pruning guided by cross-modal attention to discard task-irrelevant tokens. This holistic approach allows STAR to significantly reduce computational cost while better preserving task-critical information. Extensive experiments across multiple LVLM architectures and benchmarks show that STAR achieves strong acceleration while maintaining comparable, and in some cases even improved performance.
Paper Structure (20 sections, 13 equations, 4 figures, 3 tables)

This paper contains 20 sections, 13 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Large Vision-Language Models (LVLMs)
  4. Token Reduction for LVLMs
  5. Motivation
  6. Preliminaries
  7. Does Image Token Pruning Necessarily Lead to Visual Information Loss?
  8. Which Layer to Prune?
  9. Method
  10. Visual Self-Attention
  11. Cross-Modal Attention
  12. Theoretical Computation Reduction
  13. Experiment
  14. Experimental Setup
  15. Datasets
  16. ...and 5 more sections

Figures (4)

  • Figure 1: The framework of STAR. The two-stage pruning includes visual self-attention pruning to remove redundant tokens, followed by cross-modal attention pruning to discard task-irrelevant ones.
  • Figure 2: (a) Model performance under different pruning ratio of image tokens before the LLM decoder. (b) Impact of pruning layer and image tokens retain ratio on model performance.
  • Figure 3: The implementation of STAR mechanism. In the first stage, an importance score indicator is computed to guide the removal of low-level redundant features. In the second stage, we use the cross-modal importance indicator to select important visual tokens, which is strongly aligned to input query and generated response.
  • Figure 4: Visualization of the STAR method: Examples generated using the Vicuna-7B model