Taming the Fragility of KV Cache Eviction in LLM Inference

TL;DR

This paper tackles the memory and runtime burden of KV cache eviction in LLM inference by challenging the stability-based mean-aggregation assumption. It introduces Defensive Aggregation, a worst-case risk–aware two-step framework comprising worst-case risk estimation and adaptive prior-risk correction, and implements it in DefensiveKV and Layer-DefensiveKV. Across seven task domains and 18 datasets, these methods substantially reduce generation-quality loss at low cache budgets (e.g., 20%), outperforming state-of-the-art baselines with up to 4× greater robustness to extreme cases. The approach maintains linear-time complexity, incurs negligible overhead, and remains compatible with other efficiency techniques like quantization, marking a practical advancement for robust, memory-efficient LLM inference.

Abstract

Large language models have revolutionized natural language processing, yet their deployment remains hampered by the substantial memory and runtime overhead of the transformer's Key-Value cache. To mitigate this, recent methods employ a scoring-aggregation framework to evict unimportant cache entries, based on the stability assumption-that a fixed subset of entries remains consistently important during generation. However, prior work has largely focused on refining importance indicators for scoring, while defaulting to mean aggregation due to a faithful trust in the stability assumption. In this work, we argue that this underlying assumption is inherently fragile, making mean aggregation highly vulnerable in extreme cases. To counter this, we propose a simple yet elegant defensive aggregation strategy: a two-step, linear-time approach that controls worst-case risk, thereby defending against extreme cases with negligible computational overhead. Embodying this strategy, we propose a novel cache eviction method, DefensiveKV and its extension, Layer-DefensiveKV, which incorporates layer-wise budget allocation. Across seven task domains (18 datasets), our methods reduce generation quality loss by 2.3x and 4.3x respectively, versus the strongest baseline under a 20% cache size. These results set new performance benchmarks and pioneer a promising direction for optimizing cache eviction against underlying fragility through worst-case risk management. Our code is available at https://github.com/FFY0/DefensiveKV.

Figures (13)

• Figure 1: Defensive aggregation demonstrates robustness against fragile stability assumption (Llama-3.1-8B, 50% cache size, layer 14, summary task). Appendix \ref{['apdx:more_main']} provides additional visualizations.
• Figure 2: DefensiveKV and Layer-DefensiveKV achieve significantly lower losses of generation quality compared to all baselines across various domains and models.
• Figure 3: Mean vs. Defensive Aggregation: Against Fragility in Importance Stability
• Figure 4: Overview of averaged generation quality across 16 datasets on LongBench.
• Figure 5: Analysis of the six task domains on LongBench for Meta-Llama-3.1-8B-Instruct.