The Pitfalls of KV Cache Compression

TL;DR

The paper tackles the nonuniform degradation and leakage risks of KV cache compression in LLMs, especially in multi-instruction prompts and system prompts. It analyzes multiple eviction policies ( StreamingLLM, H2O, K-norm, SnapKV, TOVA ) across models to show that degradation, leakage, and instruction-following fidelity depend on policy, instruction order, and eviction bias. It proposes two practical remedies—token whitelisting of must-retained tokens and fair eviction that distributes retention across instruction partitions—demonstrating improvements in defending against leakage with minimal impact on directive following. These findings highlight that careful eviction policy design is essential for robust KV cache compression in realistic, multi-instruction scenarios.

Abstract

KV cache compression promises increased throughput and efficiency with negligible loss in performance. While the gains in throughput are indisputable and recent literature has indeed shown minimal degradation on particular benchmarks, in general the consequences of compression in realistic scenarios such as multi-instruction prompting have been insufficiently studied. In this paper, we identify several pitfalls practitioners should be aware of when deploying KV cache compressed LLMs. Importantly, we show that certain instructions degrade much more rapidly with compression, effectively causing them to be completely ignored by the LLM. As a practical example of that, we highlight system prompt leakage as a case study, empirically showing the impact of compression on leakage and general instruction following. We show several factors that play a role in prompt leakage: compression method, instruction order, and KV eviction bias. We then propose simple changes to KV cache eviction policies that can reduce the impact of these factors and improve the overall performance in multi-instruction tasks.

Figures (15)

• Figure 1: Existing eviction policies are unfair in multi-instruction prompts. Standard eviction policies cause certain instructions to be evicted more than others, leading to these being ignored. We propose that eviction policies should be fair w.r.t. instructions.
• Figure 2: Llama3 + StreamingLLM degradation rates for each instruction class in single- (left) and multi-instruction (right) prompts. How much the performance of each instruction class degrades is roughly described by the slope of each curve. Notably, degradation is not homogenous: each class presents a different behavior.
• Figure 3: Single- vs multi-instruction rank correlation coefficients. Spearman correlation coefficients are shown as solid lines. Coefficients closer to one indicate rankings are more similar.
• Figure 4: Both eviction policy and model play a role in performance degradation. The two plots on the left show average accuracy (across all instruction classes) on IFEval and their degradation as more compression is applied. The two plots on the right show how similar the performance (in terms of ranking) of each instruction class behaves compared to its baseline uncompressed ranking.
• Figure 5: Directive following and leakage as a function of the compression ratio. The two plots on the left show the average accuracy of directive following across all instruction classes. The two plots on the right show the ROUGE-L similarity score of the responses to the directive in the system prompt when querying for the system prompt.