DeltaKV: Residual-Based KV Cache Compression via Long-Range Similarity

TL;DR

DeltaKV addresses the memory and throughput bottlenecks of long-context KV caches by exploiting global long-range similarity and shared latent components to encode residuals relative to retrieved references. The framework combines a strided-reference residual compressor with a lightweight decompressor and a hybrid loss that preserves next-token prediction, enabling near-lossless KV reconstruction while dramatically reducing memory. When integrated with Sparse-vLLM and sparse attention (e.g., OmniKV), DeltaKV delivers substantial practical gains, reducing KV cache to about $29\%$ of original and achieving up to $2\times$ decoding throughput on long contexts such as $128k$ tokens, with continued improvements at larger contexts. The approach offers actionable insights into global redundancy in KV caches, and demonstrates a viable path to scalable, efficient long-context LLM deployment through compression- and sparsity-aware system design.

Abstract

The deployment of efficient long-context LLMs in applications like autonomous agents, long-chain reasoning, and creative writing is fundamentally bottlenecked by the linear growth of KV cache memory. Existing compression and eviction methods often struggle to balance accuracy, compression ratio, and hardware efficiency. We propose DeltaKV, a residual-based KV cache compression framework motivated by two empirical findings: long-range inter-token similarity and highly shared latent components in KV representations. Instead of discarding tokens, DeltaKV encodes semantic residuals relative to retrieved historical references, preserving fidelity while substantially reducing storage. To translate compression gains into real system speedups, we further introduce Sparse-vLLM, a high-performance inference engine with decoupled memory management and kernels optimized for sparse and irregular KV layouts. Experiments show that DeltaKV reduces KV cache memory to 29\% of the original while maintaining near-lossless accuracy on LongBench, SCBench, and AIME. When integrated with Sparse-vLLM, it achieves up to 2$\times$ throughput improvement over vLLM in long-context scenarios, demonstrating a practical path toward scalable long-context LLM deployment. Code, model checkpoints, and datasets are available at https://github.com/CURRENTF/Sparse-vLLM.

Figures (7)

• Figure 1: Illustration of two empirical observations in KV caches:long-range inter-token similarity beyond local context and highly shared latent components across KV representations.
• Figure 2: Experimental analysis of KV redundancy and residualization.
• Figure 3: Overview of DeltaKV. DeltaKV compresses KV caches by encoding token-wise residuals relative to globally retrieved reference tokens, trained with a hybrid MSE+NTP objective (top). At inference, DeltaKV integrates with sparse attention (e.g., OmniKV), storing compressed residuals and reconstructing only selected tokens on demand, reducing memory usage while preserving accuracy (bottom).
• Figure 4: Analysis of compressed KV cache values. (a) The heatmap validates a highly uniform distribution across tokens and channels; (b) The histogram reveals concentration near 0.
• Figure 5: Ablation studies on model configurations.