R-KV: Redundancy-aware KV Cache Compression for Reasoning Models

TL;DR

R-KV tackles the memory burden of long chain-of-thought reasoning by targeting redundancy in decoding-time KV caches. It combines attention-based importance scoring with a semantic-redundancy estimator to jointly select non-redundant, informative tokens for retention, while aggressively evicting repetitive content. The method achieves near FullKV performance using only 10–34% of the KV cache and up to 105% at around 16% budget on math-reasoning tasks, with substantial memory savings (~90%) and 6.6x end-to-end throughput gains. Importantly, R-KV is training-free and model-agnostic, making it practical for deployment in RL rollouts and real-time serving of reasoning LLMs. The results demonstrate strong gains over existing KV cache baselines across multiple models and datasets, and provide guidance on hyperparameter choices and efficiency considerations for long-context reasoning workloads.

Abstract

Reasoning models have demonstrated impressive performance in self-reflection and chain-of-thought reasoning. However, they often produce excessively long outputs, leading to prohibitively large key-value (KV) caches during inference. While chain-of-thought inference significantly improves performance on complex reasoning tasks, it can also lead to reasoning failures when deployed with existing KV cache compression approaches. To address this, we propose Redundancy-aware KV Cache Compression for Reasoning models (R-KV), a novel method specifically targeting redundant tokens in reasoning models. Our method preserves nearly 100% of the full KV cache performance using only 10% of the KV cache, substantially outperforming existing KV cache baselines, which reach only 60% of the performance. Remarkably, R-KV even achieves 105% of full KV cache performance with 16% of the KV cache. This KV-cache reduction also leads to a 90% memory saving and a 6.6X throughput over standard chain-of-thought reasoning inference. Experimental results show that R-KV consistently outperforms existing KV cache compression baselines across two mathematical reasoning datasets.

Figures (7)

• Figure 1: R-KV: (1) Decoding-Time Compression (§\ref{['subsection:decoding_time_compression']}); (2) KV Cache Selection with Importance and Redundancy Estimation (§\ref{['subsection:importance_scoring']}, §\ref{['subsection:redundency_estimation']}) ; (3) KV Cache Compression by joint selection (§\ref{['subsection:joint']}).
• Figure 2: Comparison of generation length and average 1-/2-gram frequency for reasoning models and ground truth of MATH-500 hendrycks2021measuring and AIME 2024 AIME2024. . Reasoning models generate substantially longer responses with 8-14$\times$ more tokens, and show higher word repetition with 5-7$\times$ higher frequency.
• Figure 3: KV selected by SnapKV. SnapKV suffers from redundancy in reasoning models. Black tokens are not selected by SnapKV; brighter colors reflect higher attention scores. Blue tokens are omitted output.
• Figure 4: Results of R-KV compared with SnapKV and FullKV on the MATH-500 and AIME24 datasets for R1-Llama-8B (top) and R1-Qwen-14B (bottom). Results are reported as pass@1 based on 64 generated responses per question.
• Figure 5: KV sekection score comparison of attention-only metric v.s. redundency-only metric v.s. R-KV with different $\lambda$. When $\lambda \geq 0.1$, the selection score starts to be dominated by attention score.