Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ForesightKV: Optimizing KV Cache Eviction for Reasoning Models by Learning Long-Term Contribution

Zican Dong, Peiyu Liu, Junyi Li, Zhipeng Chen, Han Peng, Shuo Wang, Wayne Xin Zhao

TL;DR

ForesightKV tackles the memory and compute overhead of KV caches in long-context reasoning by learning long-term KV contribution. It uses a two-stage training pipeline—Golden Eviction for supervised labels and MD P reinforcement learning with the GRPO algorithm—to guide dynamic eviction under budget $B$ and eviction interval $L$. The Golden Eviction algorithm identifies future-important KV blocks via attention scores, while the RL stage mitigates performance drops on low-entropy tokens. Across AIME2024/2025 benchmarks, ForesightKV achieves superior reasoning accuracy at half the KV budget and shows strong efficiency and cross-domain generalization.

Abstract

Recently, large language models (LLMs) have shown remarkable reasoning abilities by producing long reasoning traces. However, as the sequence length grows, the key-value (KV) cache expands linearly, incurring significant memory and computation costs. Existing KV cache eviction methods mitigate this issue by discarding less important KV pairs, but often fail to capture complex KV dependencies, resulting in performance degradation. To better balance efficiency and performance, we introduce ForesightKV, a training-based KV cache eviction framework that learns to predict which KV pairs to evict during long-text generations. We first design the Golden Eviction algorithm, which identifies the optimal eviction KV pairs at each step using future attention scores. These traces and the scores at each step are then distilled via supervised training with a Pairwise Ranking Loss. Furthermore, we formulate cache eviction as a Markov Decision Process and apply the GRPO algorithm to mitigate the significant language modeling loss increase on low-entropy tokens. Experiments on AIME2024 and AIME2025 benchmarks of three reasoning models demonstrate that ForesightKV consistently outperforms prior methods under only half the cache budget, while benefiting synergistically from both supervised and reinforcement learning approaches.

ForesightKV: Optimizing KV Cache Eviction for Reasoning Models by Learning Long-Term Contribution

TL;DR

ForesightKV tackles the memory and compute overhead of KV caches in long-context reasoning by learning long-term KV contribution. It uses a two-stage training pipeline—Golden Eviction for supervised labels and MD P reinforcement learning with the GRPO algorithm—to guide dynamic eviction under budget and eviction interval . The Golden Eviction algorithm identifies future-important KV blocks via attention scores, while the RL stage mitigates performance drops on low-entropy tokens. Across AIME2024/2025 benchmarks, ForesightKV achieves superior reasoning accuracy at half the KV budget and shows strong efficiency and cross-domain generalization.

Abstract

Recently, large language models (LLMs) have shown remarkable reasoning abilities by producing long reasoning traces. However, as the sequence length grows, the key-value (KV) cache expands linearly, incurring significant memory and computation costs. Existing KV cache eviction methods mitigate this issue by discarding less important KV pairs, but often fail to capture complex KV dependencies, resulting in performance degradation. To better balance efficiency and performance, we introduce ForesightKV, a training-based KV cache eviction framework that learns to predict which KV pairs to evict during long-text generations. We first design the Golden Eviction algorithm, which identifies the optimal eviction KV pairs at each step using future attention scores. These traces and the scores at each step are then distilled via supervised training with a Pairwise Ranking Loss. Furthermore, we formulate cache eviction as a Markov Decision Process and apply the GRPO algorithm to mitigate the significant language modeling loss increase on low-entropy tokens. Experiments on AIME2024 and AIME2025 benchmarks of three reasoning models demonstrate that ForesightKV consistently outperforms prior methods under only half the cache budget, while benefiting synergistically from both supervised and reinforcement learning approaches.
Paper Structure (45 sections, 1 theorem, 18 equations, 4 figures, 10 tables)

This paper contains 45 sections, 1 theorem, 18 equations, 4 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Empirical Study
  3. Complex Attention Patterns During Long-Text Reasoning
  4. Large Shifts in Loss for Low-Entropy Tokens
  5. ForesightKV
  6. Dynamic KV Cache Eviction During Long-Context Generation
  7. Supervised Training
  8. Golden Eviction.
  9. Training.
  10. Effectiveness of Golden Eviction.
  11. Reinforcement Learning
  12. Experiments
  13. Experiments Setup
  14. Evaluated Models and Benchmarks.
  15. Baselines.
  16. ...and 30 more sections

Key Result

Proposition 1.1

(Error Bound). Assuming the value vectors are bounded by $\|\mathbf{v}_i\|_2 \le C$, the approximation error $\|\mathbf{o}_t - \hat{\mathbf{o}}_t\|_2$ is bounded by a term strictly proportional to the evicted mass $\epsilon_t$:

Figures (4)

  • Figure 1: KV patterns in Qwen3-4B, including three patterns: (A) Global; (B) Position-dependent; (C) and (D) Semantic-dependent.
  • Figure 2: Overview of ForesightKV. ForesightKV uses a scoring model to guide dynamic KV cache eviction during long-context generations, which is trained through supervised learning and then reinforcement learning. The dashed box indicates pooling.
  • Figure 3: Comparison of ForesightKV with other KV cache eviction methods on reasoning tasks.
  • Figure 4: Change of losses with different training steps.

Theorems & Definitions (2)

  • Proposition 1.1
  • proof