Mixing Importance with Diversity: Joint Optimization for KV Cache Compression in Large Vision-Language Models

TL;DR

The paper addresses the memory bottleneck of KV caches in large vision-language models by revealing cross-modality and head-wise redundancy not captured by traditional importance-based compression. It introduces MixKV, a head-wise adaptive mixing framework that leverages per-head redundancy to balance KV importance and diversity, producing a comprehensive score for selective eviction. Through extensive experiments across diverse LVLMs and benchmarks, MixKV yields consistent gains over baseline compression methods (e.g., an average of $5.1\%$ at budget $B=64$ across multi-modal tasks and up to $+8.0$ to $+9.0$ percentage points on GUI grounding tasks) while maintaining inference efficiency, and it generalizes to LLMs. The approach offers a practical, plug-and-play improvement for long-context multi-modal understanding with scalable memory efficiency and broad applicability to real-world AI systems.

Abstract

Recent large vision-language models (LVLMs) demonstrate remarkable capabilities in processing extended multi-modal sequences, yet the resulting key-value (KV) cache expansion creates a critical memory bottleneck that fundamentally limits deployment scalability. While existing KV cache compression methods focus on retaining high-importance KV pairs to minimize storage, they often overlook the modality-specific semantic redundancy patterns that emerge distinctively in multi-modal KV caches. In this work, we first analyze how, beyond simple importance, the KV cache in LVLMs exhibits varying levels of redundancy across attention heads. We show that relying solely on importance can only cover a subset of the full KV cache information distribution, leading to potential loss of semantic coverage. To address this, we propose \texttt{MixKV}, a novel method that mixes importance with diversity for optimized KV cache compression in LVLMs. \texttt{MixKV} adapts to head-wise semantic redundancy, selectively balancing diversity and importance when compressing KV pairs. Extensive experiments demonstrate that \texttt{MixKV} consistently enhances existing methods across multiple LVLMs. Under extreme compression (budget=64), \texttt{MixKV} improves baseline methods by an average of \textbf{5.1\%} across five multi-modal understanding benchmarks and achieves remarkable gains of \textbf{8.0\%} and \textbf{9.0\%} for SnapKV and AdaKV on GUI grounding tasks, all while maintaining comparable inference efficiency. Furthermore, \texttt{MixKV} extends seamlessly to LLMs with comparable performance gains. Our code is available at \href{https://github.com/xuyang-liu16/MixKV}{\textcolor{citeblue}{https://github.com/xuyang-liu16/MixKV}}.

Figures (6)

• Figure 1: Visualization of KV cache redundancy across different models.(a) presents the similarity of keys across the same layer and head, with Qwen2-VL (bottom) processing vision-language information and Qwen2 (top) handling pure text information. (b) quantifies the average similarity distribution of keys (top) and values (bottom) across all layers and heads for Qwen2 and Qwen2-VL.
• Figure 2: Visualization of KV cache redundancy across different heads in LLMs.(a) and (b) present the average cosine similarity across heads of LLaVA-NeXT-Mistral-7B and Qwen2-VL-7B within the LLMs, with brighter heads indicating greater similarity in semantic information.
• Figure 3: t-SNE visualization of KV cache distributions under different settings. "Full KV" represents the original KV distribution of Qwen2-VL without compression, serving as the reference distribution.
• Figure 4: Efficiency comparisons of total latency and peak memory. For a context length of 32,000, "Full KV" refers to caching the entire sequence, whereas KV compression strategies employ a budget of 64. The upper part is total time, while the lower part is peak memory.
• Figure 5: More efficiency comparisons of total latency and peak memory. The top two rows of bars correspond to a budget of 128, while the bottom two rows correspond to a budget of 256.