Fast KVzip: Efficient and Accurate LLM Inference with Gated KV Eviction

TL;DR

Fast KVzip tackles the growing memory bottleneck of KV caches in long-context LLM inference by introducing a gating-based eviction mechanism. It leverages a lightweight, low-rank sink-attention gate that predicts KV importance from input hidden states, with gate training performed offline using reconstruction-derived targets while keeping the LLM frozen. The approach achieves near-lossless performance at substantial KV budget reductions (30-40%) across diverse models and tasks, and reduces prefill latency and peak memory relative to prior methods like KVzip. Its results demonstrate broad applicability to retrieval, code understanding, and reasoning tasks, and its design remains compatible with quantization and sliding-window attention, enabling practical deployment. Overall, Fast KVzip provides a general, task-agnostic, and hardware-friendly solution for efficient LLM inference in long-context scenarios.

Abstract

Efficient key-value (KV) cache management is crucial for the practical deployment of large language models (LLMs), yet existing compression techniques often incur a trade-off between performance degradation and computational overhead. We propose a novel gating-based KV cache eviction method for frozen-weight LLMs that achieves high compression ratios with negligible computational cost. Our approach introduces lightweight sink-attention gating modules to identify and retain critical KV pairs, and integrates seamlessly into both the prefill and decoding stages. The proposed gate training algorithm relies on forward passes of an LLM, avoiding expensive backpropagation, while achieving strong task generalization through a task-agnostic reconstruction objective. Extensive experiments across the Qwen2.5-1M, Qwen3, and Gemma3 families show that our method maintains near-lossless performance while evicting up to 70% of the KV cache. The results are consistent across a wide range of tasks, including long-context understanding, code comprehension, and mathematical reasoning, demonstrating the generality of our approach.

Figures (19)

• Figure 1: Highlighted results. (a) Evaluation on the KVPress benchmark kvpress (RULER-4K dataset, Qwen3-8B) compared to state-of-the-art methods from December 2025. (b) Chunked-prefill efficiency for 170K tokens using Qwen2.5-14B-1M with 30% KV cache budget and BF16 precision.
• Figure 2: Computational flow. (a) Illustration of gating at each attention layer. (b) During each forward pass, we jointly calculate importance scores and evict low-importance KV pairs. For decoding, we maintain a small buffer to cache recent hidden states, performing gating and eviction in parallel once the buffer is full. This parallelized computation reduces latency overhead. We utilize a chunk size of 16K for prefill and a buffer size of 128 tokens for decoding; at each eviction step, we retain the KV pairs of the most recent tokens, keeping 4K tokens for prefill and 128 tokens for decoding.
• Figure 3: Gating formulation of existing KV compression methods at each attention layer. Note that we compare the core components of baseline methods, as each method jointly uses other techniques. For instance, FastGen combines its approach with cumulative attention scores h2o.
• Figure 4: Gate training process. We compute the importance scores of KV pairs for a given text input following KVzip, which calculates the maximum attention score each KV pair receives during the context reconstruction kvzip. We train gating modules to predict these scores from the input hidden states at each attention layer. Please refer to \ref{['appendix:details']} for further details and the training hyperparameters.
• Figure 5: Effects of target scores for gate training, derived from reconstruction, next-token prediction, and instruction-based QA tasks. We evaluate Qwen2.5-7B-1M on a synthetic key retrieval task from SCBench scbench and the SQuAD QA task squad.