AKVQ-VL: Attention-Aware KV Cache Adaptive 2-Bit Quantization for Vision-Language Models

TL;DR

Vision-language models suffer from memory-heavy multimodal KV caches as inputs grow longer. The paper introduces AKVQ-VL, which leverages attention-driven saliency patterns—Text-Salient Attention (TSA) and Pivot-Token-Salient Attention (PSA)—to identify salient tokens and performs adaptive per-token quantization, aided by Walsh-Hadamard Transform (WHT) to construct outlier-free KV caches. Across 12 long-context multimodal tasks and multiple VLMs, AKVQ-VL maintains or improves accuracy while delivering substantial efficiency gains, including a $2.13\times$ reduction in peak memory, up to $3.25\times$ larger batch sizes, and $2.46\times$ higher throughput, outperforming LLM-oriented quantization baselines. This work enables scalable multimodal inference with very low-bit KV caches, offering practical impact for deploying VLMs in resource-constrained settings.

Abstract

Vision-language models (VLMs) show remarkable performance in multimodal tasks. However, excessively long multimodal inputs lead to oversized Key-Value (KV) caches, resulting in significant memory consumption and I/O bottlenecks. Previous KV quantization methods for Large Language Models (LLMs) may alleviate these issues but overlook the attention saliency differences of multimodal tokens, resulting in suboptimal performance. In this paper, we investigate the attention-aware token saliency patterns in VLM and propose AKVQ-VL. AKVQ-VL leverages the proposed Text-Salient Attention (TSA) and Pivot-Token-Salient Attention (PSA) patterns to adaptively allocate bit budgets. Moreover, achieving extremely low-bit quantization requires effectively addressing outliers in KV tensors. AKVQ-VL utilizes the Walsh-Hadamard transform (WHT) to construct outlier-free KV caches, thereby reducing quantization difficulty. Evaluations of 2-bit quantization on 12 long-context and multimodal tasks demonstrate that AKVQ-VL maintains or even improves accuracy, outperforming LLM-oriented methods. AKVQ-VL can reduce peak memory usage by 2.13x, support up to 3.25x larger batch sizes and 2.46x throughput.

Figures (9)

• Figure 1: Visualization of representative attention patterns in LLaMAtouvron2023llama and LLaVAliu2024visual. In the initial layers, VLM prioritize text tokens, exhibiting Text-Salient Attention (TSA). In the subsequent layers, TSA diminishes, transitioning to Pivot-Token-Salient Attention (PSA), where only a few pivot tokens dominate attention.
• Figure 2: AKVQ-VL uses an attention-aware technique to identify salient tokens and adaptively quantize the KV cache, with WHT-based equivalent transformations effectively reducing outliers of KV cache.
• Figure 3: Visualization of average attention scores in LLaVAliu2024visual, averaged across every 8 tokens. The first two layers exhibit the TSA pattern, while other layers display the PSA pattern.
• Figure 4: Visualization of average attention scores for tokens from different modalities. To mitigate the influence of sink tokens, we exclude the first five tokens. For LLM, we apply the same grouping of indices used in VLM.
• Figure 5: The magnitudes of Keys before and after WHT in LLaVAliu2024visual, layer 10, head 0.