Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A-VL: Adaptive Attention for Large Vision-Language Models

Junyang Zhang, Mu Yuan, Ruiguang Zhong, Puhan Luo, Huiyou Zhan, Ningkang Zhang, Chengchen Hu, Xiangyang Li

TL;DR

Large Vision-Language Models incur high memory and latency due to long token sequences, especially from high-resolution images. A-VL introduces per-modality adaptive attention that partitions vision caches into core/secondary/minor sets and applies a text-attention-based eviction strategy, with a lightweight, plug-and-play deployment that requires no fine-tuning. The method demonstrates near-lossless performance while substantially reducing KV-cache memory and decoder latency across three vision-language tasks and multiple LVLMs, outperforming FastV and H$_2$O in several settings. The practical impact is enabling more efficient LVLM deployment with substantial speedups, aided by a CUDA-accelerated operator for selective-attention computation. Overall, A-VL reveals distinct attention patterns across modalities in LVLMs and offers a scalable path to efficient inference without retraining.

Abstract

The Large Vision-Language Model (LVLM) integrates computer vision and natural language processing techniques, offering substantial application potential. However, these models demand extensive resources during inference. Adaptive attention techniques can dynamically reduce computational redundancy and thus improve efficiency. Although current adaptive attention methods significantly reduce the memory requirements of Transformer-based language models, they are not tailored for LVLMs. We observe that LVLMs generate responses from both remote image tokens and local text tokens, and different modalities have different attention patterns. This observation inspires us to manage the attention for each modality separately. Specifically, for visual input, we store the cache of potentially useful information but only compute the most critical parts. For language input, we care more about local information. Based on our observation and analysis of vision-language attention patterns, we develop A-VL, a plug-and-play adaptive attention tailored for LVLM inference. Extensive evaluations on three vision-language tasks and five datasets show the effectiveness of our designs. Our approach A-VL outperforms existing adaptive attention methods in reducing memory usage and computational load without compromising performance.

A-VL: Adaptive Attention for Large Vision-Language Models

TL;DR

Large Vision-Language Models incur high memory and latency due to long token sequences, especially from high-resolution images. A-VL introduces per-modality adaptive attention that partitions vision caches into core/secondary/minor sets and applies a text-attention-based eviction strategy, with a lightweight, plug-and-play deployment that requires no fine-tuning. The method demonstrates near-lossless performance while substantially reducing KV-cache memory and decoder latency across three vision-language tasks and multiple LVLMs, outperforming FastV and HO in several settings. The practical impact is enabling more efficient LVLM deployment with substantial speedups, aided by a CUDA-accelerated operator for selective-attention computation. Overall, A-VL reveals distinct attention patterns across modalities in LVLMs and offers a scalable path to efficient inference without retraining.

Abstract

The Large Vision-Language Model (LVLM) integrates computer vision and natural language processing techniques, offering substantial application potential. However, these models demand extensive resources during inference. Adaptive attention techniques can dynamically reduce computational redundancy and thus improve efficiency. Although current adaptive attention methods significantly reduce the memory requirements of Transformer-based language models, they are not tailored for LVLMs. We observe that LVLMs generate responses from both remote image tokens and local text tokens, and different modalities have different attention patterns. This observation inspires us to manage the attention for each modality separately. Specifically, for visual input, we store the cache of potentially useful information but only compute the most critical parts. For language input, we care more about local information. Based on our observation and analysis of vision-language attention patterns, we develop A-VL, a plug-and-play adaptive attention tailored for LVLM inference. Extensive evaluations on three vision-language tasks and five datasets show the effectiveness of our designs. Our approach A-VL outperforms existing adaptive attention methods in reducing memory usage and computational load without compromising performance.
Paper Structure (25 sections, 2 equations, 10 figures, 3 tables)

This paper contains 25 sections, 2 equations, 10 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Large Vision-Language Modal.
  4. Inference Optimization for LLM and LVLM.
  5. Observations and Insights
  6. Attention of Different Modalities
  7. Heterogeneity of Vision Attention
  8. Continuity of Vision Attention
  9. Locality of Text Attention
  10. A-VL Design
  11. Adaptive Text-Aware Vision Attention
  12. Decode phase.
  13. Prefill phase.
  14. Adaptive Text Attention
  15. Implementation of A-VL
  16. ...and 10 more sections

Figures (10)

  • Figure 1: Variations in attention scores among different types of tokens during the decode phase.
  • Figure 2: The token sequence is composed of encoded inputs from different modalities.
  • Figure 3: The vision attention during the generation of new tokens is very sparse.
  • Figure 4: The correlation of attention between the first decoder layer and each subsequent decoder layer.
  • Figure 5: The correlation of attention between the first step and each subsequent step in the same decoder layer.
  • ...and 5 more figures