G-KV: Decoding-Time KV Cache Eviction with Global Attention

TL;DR

G-KV addresses the memory and compute bottlenecks of long reasoning in LLMs by introducing a global attention-based KV-cache eviction score that combines current local attention with historical significance. The approach, together with post-training strategies (reinforcement learning and distillation), enables the model to adapt to compressed KV caches, yielding substantial improvements in pass@1 on math and coding tasks under tight KV budgets. Empirical results show state-of-the-art gains across benchmarks and budgets, with notable memory reductions and throughput increases, especially for long-generation scenarios. The work provides a practical, training-aware framework to enable efficient, scalable reasoning in large models, with publicly available code for replication.

Abstract

Recent reasoning large language models (LLMs) excel in complex tasks but encounter significant computational and memory challenges due to long sequence lengths. KV cache compression has emerged as an effective approach to greatly enhance the efficiency of reasoning. However, existing methods often focus on prompt compression or token eviction with local attention score, overlooking the long-term importance of tokens. We propose G-KV, a KV cache eviction method that employs a global scoring mechanism, combining local and historical attention scores to more accurately assess token importance. Additionally, we introduce post-training techniques, including reinforcement learning and distillation, to optimize models for compressed KV cache settings. The code of this paper is available on: https://github.com/microsoft/G-KV.

Figures (24)

• Figure 1: The left figure illustrates the calculation process for overlap. The right figure depicts the overlap between the last window and other windows across different layers. The horizontal axis represents the proportion of tokens retained.
• Figure 2: This figure illustrates the computation process of the global score. Each block represents the KV cache of a token, with the block's color indicating its score (darker color represents higher scores).
• Figure 3: Illustration of the sparse attention mask. If the block at the $i$-th row and $j$-th column is visible, it indicates that the $j$-th token in the sequence can attend to the $i$-th token. Red crosses represent tokens evicted during a KV cache compression process; these tokens are invisible to newly generated tokens in subsequent steps.
• Figure 4: Performance of different methods on the AMC 23 benchmark.
• Figure 5: Performance of different compression methods with DeepSeek-R1-Distill Qwen 7B model.