Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient LLM Inference with Kcache

Qiaozhi He, Zhihua Wu

TL;DR

KCache addresses the memory bottleneck of KV Cache during autoregressive LLM inference by keeping K states in high-bandwidth memory and moving V states to CPU RAM, retrieving V vectors on-demand via a TopN-filtered attention mechanism. The method is training-free and aims to maximize throughput for long-context inputs while preserving accuracy. Through extensive experiments on multiple decoder-only models and benchmarks, KCache demonstrates around forty percent throughput gains in long-context regimes with minimal accuracy loss, validating its practical impact for scalable inference. The work highlights flexible memory management across HBM and CPU, offering a viable path to efficient, cost-effective LLM deployment in production settings.

Abstract

Large Language Models(LLMs) have had a profound impact on AI applications, particularly in the domains of long-text comprehension and generation. KV Cache technology is one of the most widely used techniques in the industry. It ensures efficient sequence generation by caching previously computed KV states. However, it also introduces significant memory overhead. We discovered that KV Cache is not necessary and proposed a novel KCache technique to alleviate the memory bottleneck issue during the LLMs inference process. KCache can be used directly for inference without any training process, Our evaluations show that KCache improves the throughput of popular LLMs by 40% with the baseline, while keeping accuracy.

Efficient LLM Inference with Kcache

TL;DR

KCache addresses the memory bottleneck of KV Cache during autoregressive LLM inference by keeping K states in high-bandwidth memory and moving V states to CPU RAM, retrieving V vectors on-demand via a TopN-filtered attention mechanism. The method is training-free and aims to maximize throughput for long-context inputs while preserving accuracy. Through extensive experiments on multiple decoder-only models and benchmarks, KCache demonstrates around forty percent throughput gains in long-context regimes with minimal accuracy loss, validating its practical impact for scalable inference. The work highlights flexible memory management across HBM and CPU, offering a viable path to efficient, cost-effective LLM deployment in production settings.

Abstract

Large Language Models(LLMs) have had a profound impact on AI applications, particularly in the domains of long-text comprehension and generation. KV Cache technology is one of the most widely used techniques in the industry. It ensures efficient sequence generation by caching previously computed KV states. However, it also introduces significant memory overhead. We discovered that KV Cache is not necessary and proposed a novel KCache technique to alleviate the memory bottleneck issue during the LLMs inference process. KCache can be used directly for inference without any training process, Our evaluations show that KCache improves the throughput of popular LLMs by 40% with the baseline, while keeping accuracy.
Paper Structure (11 sections, 4 equations, 2 figures, 4 tables)

This paper contains 11 sections, 4 equations, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Method
  4. KCache
  5. Analysis of KCache Performance
  6. Analysis of KCache Accuracy
  7. Experiments
  8. Setup
  9. Results
  10. Conclusion
  11. Appendix

Figures (2)

  • Figure 1: Illustration of the KCache. During $prefill$ phase, the computation results of each layer $push$ to the HBM. After that, the part of V Cache will be copied to the CPU asynchronously, while releasing the GPU memory occupied by this part of the V Cache. During $decode$ phase, $K$ states will be pushed and pulled as KV Cache. However, we will calculate the $topN$ of attention scores, and based on the indices of the topN results, we will pull the corresponding V Cache from the CPU to the HBM in real-time to complete the subsequent computation.
  • Figure 2: Prompt length of BBH, GSM8K and TriviaQA.