LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference

TL;DR

The paper presents LMCache, an open-source KV caching layer that offloads KV caches from GPU memory to enable cross-query and cross-engine reuse in enterprise-scale LLM inference. It introduces a modular connector API, batched data movement, and multi-tier storage to maximize throughput, achieving up to 15x throughput improvements and notable latency reductions with vLLM and PD disaggregation. Real-world deployments yield insights on remote storage benefits, context truncation effects, and containerized workflows, underscoring LMCache's practical impact and adaptability. The work highlights a shift toward AI-native data planes for scalable inference and offers a foundation for broader KV-cache-aware infrastructure.

Abstract

KV cache has traditionally been stored in GPU memory to accelerate the decoding phase of large language model (LLM) inference. However, it is increasingly necessary to move KV caches outside GPU devices, to enable cache reuse across different queries and inference engines. Our real-world usage statistics confirm this trend: over time, the total KV cache stored by users has grown rapidly, far exceeding the capacity of GPU memory. Despite this need, there lacks an efficient solution for offloading and transferring KV caches. We present LMCACHE, the first and so far the most efficient open-source KV caching solution, which extracts and stores KV caches generated by modern LLM engines (vLLM and SGLang) out of the GPU memory and shares them across engines and queries. LMCACHE supports both cache offloading (prefix reuse across queries) and prefill-decode (PD) disaggregation (cross-engine/GPU cache transfer). LMCACHE's high performance and wide adoption stem from the following contributions: (1) highly optimized KV cache data movement powered by batched data movement operations, compute and I/O pipelining; (2) a modular KV cache connector component, decoupling LMCACHE from the rapid evolution of inference engines; (3) a first-class control API for flexible cache orchestration across GPU, CPU, storage, and network layers. Our evaluation shows that combining LMCACHE with vLLM achieves up to 15x improvement in throughput across workloads such as multi-round question answering and document analysis. Large-scale adoption of LMCACHE in enterprise settings provides us valuable insights, for example, fetching KV cache from remote storage has unsurprisingly benefits to prefill delay, and that context truncation, which is a widely applied technique in industry, can greatly reduce prefix cache hit ratio by half. The source code of LMCACHE is at: https://github.com/LMCache/LMCache.

Figures (16)

• Figure 1: Upper: LMCache is being used by more and more users over time. Middle: LMCache is being used to store KV cache of larger size. Bottom: LMCache's docker image pull counts have continued to increase.
• Figure 2: LMCache supports both context caching (KV cache offloading and sharing across queries) and PD disaggregation (cross-engine transfer of KV caches.
• Figure 3: Weekly growth of KV cache size, including portions that fit in GPU memory and those that exceed it.
• Figure 4: Left: Average reuse per token for top users. Right: Distributions of average reuse per token across different users.
• Figure 5: LMCache sits between LLM inference engines and heterogeneous storage/network devices.