Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Hierarchical Adaptive Eviction for KV Cache Management in Multimodal Language Models

Xindian Ma, Yidi Lu, Peng Zhang, Jing Zhang

TL;DR

HAE tackles the memory and compute bottleneck of KV-cache in multimodal LLMs by introducing a hierarchical eviction framework that treats visual and textual tokens differently. It combines Dual-Attention Pruning in the pre-fill stage with a Dynamic Decoding Eviction Strategy during decoding, and provides theoretical bounds on information integrity and eviction error. Empirically, HAE reduces KV-cache memory by about 41–47% with minimal accuracy loss (~0.3%) and speeds up long-form image-based generation by approximately 1.5x, while maintaining or improving output quality on multimodal benchmarks. The approach offers a practical, training-free mechanism to improve efficiency in cross-modal reasoning and generation, with strong theoretical guarantees and broad applicability to existing MLLMs like Phi3.5-Vision-Instruct and LLaVA-1.5-7B.

Abstract

The integration of visual information into Large Language Models (LLMs) has enabled Multimodal LLMs (MLLMs), but the quadratic memory and computational costs of Transformer architectures remain a bottleneck. Existing KV cache eviction strategies fail to address the heterogeneous attention distributions between visual and text tokens, leading to suboptimal efficiency or degraded performance. In this paper, we propose Hierarchical Adaptive Eviction (HAE), a KV cache eviction framework that optimizes text-visual token interaction in MLLMs by implementing Dual-Attention Pruning during pre-filling (leveraging visual token sparsity and attention variance) and a Dynamic Decoding Eviction Strategy (inspired by OS Recycle Bins) during decoding. HAE minimizes KV cache usage across layers, reduces computational overhead via index broadcasting, and theoretically ensures superior information integrity and lower error bounds compared to greedy strategies, enhancing efficiency in both comprehension and generation tasks. Empirically, HAE reduces KV-Cache memory by 41\% with minimal accuracy loss (0.3\% drop) in image understanding tasks and accelerates story generation inference by 1.5x while maintaining output quality on Phi3.5-Vision-Instruct model.

Hierarchical Adaptive Eviction for KV Cache Management in Multimodal Language Models

TL;DR

HAE tackles the memory and compute bottleneck of KV-cache in multimodal LLMs by introducing a hierarchical eviction framework that treats visual and textual tokens differently. It combines Dual-Attention Pruning in the pre-fill stage with a Dynamic Decoding Eviction Strategy during decoding, and provides theoretical bounds on information integrity and eviction error. Empirically, HAE reduces KV-cache memory by about 41–47% with minimal accuracy loss (~0.3%) and speeds up long-form image-based generation by approximately 1.5x, while maintaining or improving output quality on multimodal benchmarks. The approach offers a practical, training-free mechanism to improve efficiency in cross-modal reasoning and generation, with strong theoretical guarantees and broad applicability to existing MLLMs like Phi3.5-Vision-Instruct and LLaVA-1.5-7B.

Abstract

The integration of visual information into Large Language Models (LLMs) has enabled Multimodal LLMs (MLLMs), but the quadratic memory and computational costs of Transformer architectures remain a bottleneck. Existing KV cache eviction strategies fail to address the heterogeneous attention distributions between visual and text tokens, leading to suboptimal efficiency or degraded performance. In this paper, we propose Hierarchical Adaptive Eviction (HAE), a KV cache eviction framework that optimizes text-visual token interaction in MLLMs by implementing Dual-Attention Pruning during pre-filling (leveraging visual token sparsity and attention variance) and a Dynamic Decoding Eviction Strategy (inspired by OS Recycle Bins) during decoding. HAE minimizes KV cache usage across layers, reduces computational overhead via index broadcasting, and theoretically ensures superior information integrity and lower error bounds compared to greedy strategies, enhancing efficiency in both comprehension and generation tasks. Empirically, HAE reduces KV-Cache memory by 41\% with minimal accuracy loss (0.3\% drop) in image understanding tasks and accelerates story generation inference by 1.5x while maintaining output quality on Phi3.5-Vision-Instruct model.
Paper Structure (25 sections, 2 theorems, 21 equations, 5 figures, 6 tables)

This paper contains 25 sections, 2 theorems, 21 equations, 5 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Observation
  4. Differences in cumulative distribution
  5. Sparse Rate for Different Layers
  6. Hierarchical Adaptive Eviction
  7. Dual-Attention Pruning
  8. Dynamic Decoding Eviction Strategy
  9. Theoretical Analysis
  10. Related Work
  11. Experiments
  12. Setup
  13. Main Results
  14. Ablation Study
  15. Pruning Visual Token Analysis
  16. ...and 10 more sections

Key Result

Theorem 2.1

(Cache Information Integrity). If the eviction threshold $k$ meets then the total loss of the evicted tokens is $\sum_{j=1}^{c} \epsilon_j < \epsilon$, where $Attn_{max} = \mathop{min}\{\mathop{max}\limits_{j} A_{j,i}|j\in[1,2,...,c], i\in[1,2,...,|T|]\}$, $\lambda$ is the decay rate, and $\epsilon$ is the allowable loss error.

Figures (5)

  • Figure 1: The overall framework of Hierarchical Adaptive Eviction (HAE). In the pre-filling stage, the Dual-Attention KV Pruning (DAP) detects redundant visual token indices 2 and 4 in the first layer, and broadcasts this information to perform corresponding eviction across other layers. In the decoding stage, the Dynamic Decoding Eviction Strategy (DDES) handles KV eviction. For example, when generating token 9, index 3 is marked but not removed; when generating token 10, index 7 is marked. Once the recycling bin is full, indices 3 and 7 are evicted, and then the indices are reset.
  • Figure 2: Comparison of cumulative attention score variance between visual tokens and text tokens in the first transformer layer of Phi3.5-vision-Instruction.
  • Figure 3: Left: Sparsity rates across layers in Phi3.5-vision-Instruct, highlighting overall, visual, and text component sparsity. Right: Attention matrix for layer 1, featuring annotated token positions that connect visual and text elements to their attention patterns.
  • Figure 4: Comparison of image-based story generation results using different KV eviction acceleration methods based on Phi3.5-Vision-Instruct model. (a) Annotated story description; (b) Results generated using the H2O method; (c) Results generated using the MustDrop method; (d) Results generated using our proposed HAE method.
  • Figure 5: Comparison of cumulative attention score variance between visual tokens and text tokens in the Phi3.5-vision-Instruction.

Theorems & Definitions (4)

  • Definition 1
  • Definition 2
  • Theorem 2.1
  • Corollary 2.1