Revisiting Multimodal KV Cache Compression: A Frequency-Domain-Guided Outlier-KV-Aware Approach

TL;DR

This work tackles the substantial memory and compute overhead of multimodal KV Cache in long-context LLMs by adopting a frequency-domain perspective. It reveals that KV energy concentrates in low-frequency components and introduces Outlier KVs—KV pairs whose deviation from a smoothed Base KV carries critical inference information. FlashCache combines an Outlier KV Recognition Module (DCT-based, low-pass Base KV estimation) with a Dynamic Budget Allocation Module (layer-wise, energy-based budgets) to achieve attention-score-free KV compression compatible with FlashAttention. Across multi-image, high-resolution, and video benchmarks, FlashCache delivers up to 1.69× decoding speedups and up to 80% reduction in KV memory while maintaining or improving task performance, highlighting practical benefits for scalable multimodal inference.

Abstract

Multimodal large language models suffer from substantial inference overhead since multimodal KV Cache grows proportionally with the visual input length. Existing multimodal KV Cache compression methods mostly rely on attention score to reduce cache size, which makes them are incompatible with established efficient attention kernels (e.g., FlashAttention) and ignores the contribution of value vectors to the attention output. In this work, we revisit multimodal KV Cache compression from the perspective of the KV matrices' distribution. First, we observe that frequency-domain energy of multimodal KV matrices is predominantly concentrated in low-frequency and extract this principal energy via a low-pass filter. Further, we find that removing KV pairs that deviate substantially from this principal energy leads to a pronounced performance drop, which we define as Outlier KVs. Considering Outlier KVs are more likely to encode features critical for inference, we propose FlashCache, a frequency-domain-guided, Outlier-KV-aware KV Cache compression framework. First, we introduce an Outlier KV Recognition Module that models the principal component of multimodal KV matrices in the frequency domain and preferentially retains KV pairs that significantly deviate from it. Furthermore, Dynamic Budget Allocation Module is designed to adaptively determine the per-layer KV Cache size to retain more Outlier KVs. Experiments on multiple MLLMs and benchmarks demonstrate that FlashCache outperforms state-of-the-art multimoal KV compression methods, achieving up to 1.69 times faster decoding with 80% lower KV memory usage while maintaining task performance.

Figures (11)

• Figure 1: Frequency-domain energy distribution of the KV matrices. We observe that the frequency-domain energy of KV matrices is predominantly concentrated at low frequency, with high frequency components occupying a relatively small proportion. The red dashed box in the figure highlights the low-frequency concentration phenomenon. Experiments are conducted on MileBench song2024milebench with Qwen2.5-VL-7B.
• Figure 2: The impact on model performance of prioritizing the removal of KV pairs under different removal ratios. We find discarding KV pairs with larger deviations leads to faster performance drop than random removal or removing small-deviation pairs. Experiments are conducted on MileBench song2024milebench with Qwen2.5-VL-7B.
• Figure 3: Overview of FlashCache. We perform compression on the multimodal KV Cache in a single operation after the prefilling stage. FlashCache comprises two core components: the Outlier KV Recognition Module and the Dynamic Budget Allocation Module. First, the Outlier KV Recognition Module obtains the primary smoothed component Base KV from the KV matrices via the Base KV Filter. Next, it prioritizes retaining KV pairs with larger deviations of each KV pair from Base KV, which are defined as Outlier KVs. The Dynamic Budget Allocation Module measures the energy intensity of outlier information at each layer to dynamic allocation of the KV Cache size for each layer. ($HSM^{l}$ represents the hidden state matrix of layer $l$’s input in the model.)
• Figure 4: Frequency-domain energy distribution of KV matrices across different layers. We observe significant differences in the frequency-domain energy distribution of KV matrices across different layers. The red dashed box indicates the energy distributions of KV matrices of different layers show different modes. Experiments are conducted on MileBench song2024milebench with Qwen2.5-VL-7B.
• Figure 5: The impact of KV Cache retention ratio changes on performance. We compare the performance impact of different KV Cache compression methods across six KV Cache retention ratio settings (0.8, 0.6, 0.4, 0.2, 0.1, 0.05), where FlashCache consistently shows superior performance, particularly under low retention ratios. The horizontal axis represents the KV Cache retention ratio.