Understanding Bottlenecks for Efficiently Serving LLM Inference With KV Offloading

TL;DR

This work analyzes KV cache offloading for long-context LLM inference and identifies a fundamental memory-bound bottleneck driven by CPU-GPU PCIe bandwidth. By deriving the critical ratio $κ_{ ext{crit}}$ and the workload ratio $κ_{ ext{ratio}}$, the authors predict when prefill shifts from compute- to memory-bound regimes and validate these insights with extensive experiments across model types and hardware platforms. They find that real workloads routinely operate well into the memory-bound region (often with up to 99% of time spent on transfers) and that GPUs run far below peak TDP, highlighting inefficiencies in current serving stacks. The paper then proposes a set of hardware, model-architecture, and workload-aware scheduling optimizations—such as MLA-based KV compression, NVLink/HBM integrations, and utilization-aware scheduling—to extend the compute-bound regime and improve overall latency and throughput for KV offloaded inference.

Abstract

KV cache offloading enables long-context LLM inference by storing caches in CPU DRAM, but PCIe bandwidth limitations create severe bottlenecks. In this paper, we develops an analytical framework that derives $κ_{\text{crit}}$, the critical cached-to-prefill token ratio where execution becomes memory-bound and show typical workloads exceed this threshold by orders of magnitude. Empirical characterization reveals 99\% of latency spent on transfers and serving offloaded requests results in GPU's consuming only 28\% of their rated TDP, motivating our proposed optimizations for hardware interconnects, model architectures, and scheduling algorithms.

Figures (8)

• Figure 1: KV cache offloading transfers computed representations from GPU VRAM to CPU DRAM. When serving the second request (green), only the novel token ("ocean") requires computation; cached tokens are loaded via PCIe.
• Figure 2: Overview of iterative scheduling in the vLLM scheduler vllm_v1
• Figure 3: Distributions of prefill tokens ($T$), reused KV tokens ($K$), and the ratio of reused KV to prefill tokens ($\kappa_{\text{ratio}}$) for ShareGPT (top), NarrativeQA (middle), and FinQA (bottom). Median Values for $\kappa_{\text{ratio}}$: ShareGPT: 100, NarrativeQA: 5000, FinQA: 10000
• Figure 4: Roofline models for Qwen3-235B-A22B (left) and Deepseek-V3 (right) on NVIDIA B200, H100 and A100 for current PCIe bandwidths. Vertical lines indicate $\kappa_{\mathrm{crit}}$ for specific hardware. Points left of $\kappa_{\mathrm{crit}}$ are compute-bound; right are bandwidth-bound.
• Figure 5: Measurements of $\kappa_{\text{crit}}$.