Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference

Xiang Liu, Zhenheng Tang, Peijie Dong, Zeyu Li, Yue Liu, Bo Li, Xuming Hu, Xiaowen Chu

TL;DR

ChunkKV tackles the memory bottleneck of KV caches in long-context LLM inference by preserving semantic chunks rather than individual tokens. It introduces a chunk-based KV cache compression alongside a training-free layer-wise index reuse to reduce overhead and improve throughput. Across LongBench, Needle-In-A-HayStack, GSM8K, and JailbreakV, ChunkKV delivers consistent accuracy gains at fixed compression ratios and outperforms state-of-the-art token-based methods, while remaining competitive with KV quantization approaches. The approach is simple, training-free, and practically impactful for deploying large language models in memory-constrained environments, with code available from NVIDIA/kvpress.

Abstract

Large Language Models (LLMs) require significant GPU memory when processing long texts, with the key value (KV) cache consuming up to 70\% of total memory during inference. Although existing compression methods reduce memory by evaluating the importance of individual tokens, they overlook critical semantic relationships between tokens, resulting in fragmented context and degraded performance. We introduce ChunkKV, which fundamentally reimagines KV cache compression by treating semantic chunks - rather than isolated tokens - as basic compression units. This approach preserves complete linguistic structures and contextual integrity, ensuring that essential meaning is retained even under aggressive compression. Our innovation includes a novel layer-wise index reuse technique that exploits the higher cross-layer similarity of preserved indices in ChunkKV, reducing computational overhead and improving throughput by 26.5\%. Comprehensive evaluations on challenging benchmarks: LongBench, Needle-In-A-HayStack, GSM8K, and JailbreakV demonstrate that ChunkKV outperforms state-of-the-art methods by up to 8.7\% in precision while maintaining the same compression ratio. These results confirm that semantic-aware compression significantly enhances both efficiency and performance for long-context LLM inference, providing a simple yet effective solution to the memory bottleneck problem. The code is available at \href{https://github.com/NVIDIA/kvpress}{link}.

ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference

TL;DR

ChunkKV tackles the memory bottleneck of KV caches in long-context LLM inference by preserving semantic chunks rather than individual tokens. It introduces a chunk-based KV cache compression alongside a training-free layer-wise index reuse to reduce overhead and improve throughput. Across LongBench, Needle-In-A-HayStack, GSM8K, and JailbreakV, ChunkKV delivers consistent accuracy gains at fixed compression ratios and outperforms state-of-the-art token-based methods, while remaining competitive with KV quantization approaches. The approach is simple, training-free, and practically impactful for deploying large language models in memory-constrained environments, with code available from NVIDIA/kvpress.

Abstract

Large Language Models (LLMs) require significant GPU memory when processing long texts, with the key value (KV) cache consuming up to 70\% of total memory during inference. Although existing compression methods reduce memory by evaluating the importance of individual tokens, they overlook critical semantic relationships between tokens, resulting in fragmented context and degraded performance. We introduce ChunkKV, which fundamentally reimagines KV cache compression by treating semantic chunks - rather than isolated tokens - as basic compression units. This approach preserves complete linguistic structures and contextual integrity, ensuring that essential meaning is retained even under aggressive compression. Our innovation includes a novel layer-wise index reuse technique that exploits the higher cross-layer similarity of preserved indices in ChunkKV, reducing computational overhead and improving throughput by 26.5\%. Comprehensive evaluations on challenging benchmarks: LongBench, Needle-In-A-HayStack, GSM8K, and JailbreakV demonstrate that ChunkKV outperforms state-of-the-art methods by up to 8.7\% in precision while maintaining the same compression ratio. These results confirm that semantic-aware compression significantly enhances both efficiency and performance for long-context LLM inference, providing a simple yet effective solution to the memory bottleneck problem. The code is available at \href{https://github.com/NVIDIA/kvpress}{link}.

Paper Structure

This paper contains 44 sections, 2 theorems, 13 equations, 20 figures, 30 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. ChunkKV
  4. Preliminary Study of KV Cache Compression
  5. Chunk Based KV Compression
  6. Layer-Wise Index Reuse
  7. Experiment Results
  8. In-Context Learning
  9. Long-Context Benchmark
  10. Index Reuse
  11. Chunk Size
  12. Comparing with KV Quantization
  13. Analysis of Hybrid Compression: Chunk-level vs. Token-level at Different Layer Depths
  14. Experimental Setup
  15. Analysis and Insights
  16. ...and 29 more sections

Key Result

Lemma 1

let $\mathcal{B}$ denotes the set of $\theta$ which does not satisfy Condition cond:distinguish. We assume that $\text{KL}(p_{prompt}(y_\text{test}|x_\text{test}))||p(y_\text{test}|x_\text{test},\theta)$ is bounded for all $\theta$ and that $\theta^\star$ minimizes the multi-class logistic risk as, If then where $g(\nu) = \frac{1}{2}((1-\nu)\log(1-\nu)+(1+\nu)\log(1+\nu))$ is the calibration fu

Figures (20)

  • Figure 1: Illustration of the impact of the token discrete method and the chunk method on semantic preservation. The discrete method preserves words related to the question but often omits the subject. In contrast, the chunk method retains the subject of the words, maintaining more accurate semantic information. For the equation: $S$ is the score function, and $c$ is a chunk of tokens.
  • Figure 2: Layer-wise similarity heatmaps of the preserved KV cache indices by SnapKV (left) and ChunkKV (right) on LLaMA-3-8B-Instruct. Deep colors indicate higher similarity. More visualization can be found in Appendix \ref{['appendix:index_reuse_similarity']}.
  • Figure 3: NIAH benchmark for LLaMA3-8B-Instruct with KV cache size=128 under 8k context length.
  • Figure 4: Layer-wise comparison of L1 loss and attention cosine similarity between ChunkKV and discrete token-based methods in Single-Document QA sub-category of LongBench.
  • Figure 5: GSM8K Performance Comparison with different index reuse layers
  • ...and 15 more figures

Theorems & Definitions (3)

  • Lemma 1: noisy-relaxed bound in xie2022anzhoucan
  • Theorem 1
  • proof