Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Joint Encoding of KV-Cache Blocks for Scalable LLM Serving

Joseph Kampeas, Emir Haleva

TL;DR

This work tackles the KV-cache memory bottleneck in scalable LLM serving by introducing joint KV-cache block encoding through Batch Fast-Fusion and Chunks Fast-Fusion. It presents a tree-based fusion strategy that shares similar cache blocks across requests or input chunks while preserving the original cache structure, enabling larger batch sizes and reducing memory, bandwidth, and network demands. A rate-distortion framework based on a Poisson process provides theoretical insight into the threshold-driven trade-off between compression and distortion, and extensive experiments show up to $4.38×$ KV-cache compression with negligible accuracy loss and substantial throughput gains on diverse models and benchmarks. The approach promisingly enhances real-world serving scalability, with practical implications for disaggregated architectures like vLLM and beyond, and points to future work in adaptive thresholds and tighter integration with quantization and pruning techniques.

Abstract

Modern large language models (LLMs) drive interactive AI systems but are bottlenecked by the memory-heavy growth of key-value (KV) caches, which limits real-time throughput under concurrent loads. Existing KV-cache compression methods rely on rigid heuristics, disrupt tensor layouts, or require specialized compute, hindering scalability and deployment. We propose joint encoding of KV-cache blocks, which fuses similar blocks across requests and input chunks into shared representations while preserving standard cache structure. This alleviates the KV-cache memory bottleneck, supporting high-concurrency serving without specialized hardware. Theoretically, we analyze the rate-distortion tradeoff of fused cache blocks under a Poisson process model. Empirically, our method achieves up to 4.38 $\times$ KV-cache compression with negligible accuracy loss across diverse LLMs and benchmarks, outperforming recent structured and adaptive compression baselines. In real LLM serving, joint encoding improves the token throughput by $\sim$40\% on a single-machine vLLM benchmark, demonstrating substantial gains in inference throughput. Code is available at https://github.com/sef1/kv_fast_fusion kv_joint_encoding.

Joint Encoding of KV-Cache Blocks for Scalable LLM Serving

TL;DR

This work tackles the KV-cache memory bottleneck in scalable LLM serving by introducing joint KV-cache block encoding through Batch Fast-Fusion and Chunks Fast-Fusion. It presents a tree-based fusion strategy that shares similar cache blocks across requests or input chunks while preserving the original cache structure, enabling larger batch sizes and reducing memory, bandwidth, and network demands. A rate-distortion framework based on a Poisson process provides theoretical insight into the threshold-driven trade-off between compression and distortion, and extensive experiments show up to KV-cache compression with negligible accuracy loss and substantial throughput gains on diverse models and benchmarks. The approach promisingly enhances real-world serving scalability, with practical implications for disaggregated architectures like vLLM and beyond, and points to future work in adaptive thresholds and tighter integration with quantization and pruning techniques.

Abstract

Modern large language models (LLMs) drive interactive AI systems but are bottlenecked by the memory-heavy growth of key-value (KV) caches, which limits real-time throughput under concurrent loads. Existing KV-cache compression methods rely on rigid heuristics, disrupt tensor layouts, or require specialized compute, hindering scalability and deployment. We propose joint encoding of KV-cache blocks, which fuses similar blocks across requests and input chunks into shared representations while preserving standard cache structure. This alleviates the KV-cache memory bottleneck, supporting high-concurrency serving without specialized hardware. Theoretically, we analyze the rate-distortion tradeoff of fused cache blocks under a Poisson process model. Empirically, our method achieves up to 4.38 KV-cache compression with negligible accuracy loss across diverse LLMs and benchmarks, outperforming recent structured and adaptive compression baselines. In real LLM serving, joint encoding improves the token throughput by 40\% on a single-machine vLLM benchmark, demonstrating substantial gains in inference throughput. Code is available at https://github.com/sef1/kv_fast_fusion kv_joint_encoding.
Paper Structure (13 sections, 3 theorems, 9 equations, 7 figures, 2 tables, 1 algorithm)

This paper contains 13 sections, 3 theorems, 9 equations, 7 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Motivation
  4. KV Blocks Joint-Encoding
  5. Analysis
  6. Results
  7. Block Diversity and Rate-Distortion
  8. Batch Fast-Fusion During Decoding
  9. Chunks Fast Fusion During Prefill
  10. End-to-End Serving Perfromance
  11. Conclusion
  12. Appendix
  13. Proof to \ref{['lem:Attention (softmax) error bound']}

Key Result

Proposition 1

For a sufficiently high similarity threshold $u$, the number of above-threshold similarities is asymptotically a Poisson variable with rate where $a_n = (2\log n)^{-1/2}$ and $b_n = (2\log n)^{1/2} - \frac{1}{2}(2 \log n)^{-1/2}(\log \log n + \log(4\pi))$ are normalization constants.

Figures (7)

  • Figure 1: Attention computation (a) batch of requests with similar keys blocks. After the fusion, the KV=cache footprint is smaller and batch decoding is optimized by performing matrix-matrix multiplication. (b) After fusing chunks into a unified representation, the blocks computation of chunks 0 and 1 can be reused in chunk 2 (same color).
  • Figure 2: Fast Fusion
  • Figure 3: Empirical similarity and analysis for DeepSeekR1DistillQwen7B on nVidiaHelpSteer dataset for several layers. Bars represents the empirical similarity and solid line the KDE approximation. (a) Blocks similarity density. (b) Above-threshold density for threshold=0.8.
  • Figure 4: CR and F1 score of BFF and CFF for Llama-2 7B. (a) CR vs. batch size for diverse number of blocks on vLLM random-data benchmark. (b) BFF F1 score vs. CR for batch size 256 on conversational dataset. (c) CFF F1 score vs. CR, for 4 chunks on conversational dataset.
  • Figure 5: CR and F1 score of BFF vs. baseline for Llama-3.1 8B and Qwen2.5 72B on nVidiaOpenMathInstruct dataset. (a) CR vs. batch size. (b) F1 score vs. batch size.
  • ...and 2 more figures

Theorems & Definitions (8)

  • Remark 1
  • Remark 2
  • Remark 3
  • Proposition 1
  • proof
  • Corollary 1
  • Lemma 1
  • proof