ContiguousKV: Accelerating LLM Prefill with Granularity-Aligned KV Cache Management

TL;DR

ContiguousKV tackles the Re-Prefill bottleneck in shared-prefix LLM serving by co-designing data granularity with pruning semantics via ContiguousChunk, enabling zero read amplification. It introduces two-level asynchronous prefetching across Periods to pipeline I/O with computation and an attention-guided cache policy to keep semantically important chunks in fast memory. Empirical results on Qwen2.5 models demonstrate up to a 3.85x speedup in Re-Prefill over state-of-the-art systems while preserving accuracy, with notable I/O reductions and tail-latency improvements. This work provides a scalable path for memory-efficient, low-latency shared-prefix inference applicable to conversational search, RAG, and long-context tasks.

Abstract

Efficiently serving Large Language Models (LLMs) with persistent Prefix Key-Value (KV) Cache is critical for applications like conversational search and multi-turn dialogue. Serving a request requires loading the pre-computed prefix KV cache and generating the first token, defined as the Re-Prefill Phase. Offloading this shared prefix cache to secondary storage is essential for memory scalability. Re-Prefill with offloading suffers from severe I/O bottlenecks in two aspects. First, semantic-aware KV cache pruning algorithms select important tokens in fine granularity, while systems manage I/O in coarse, fixed-size blocks, causing severe read amplification. Second, the sequential dependency between identifying important tokens and loading KV cache creates idle I/O and compute bubbles, under-utilizing system resources. This paper proposes \textit{ContiguousKV}, a high-performance prefix KV cache offloading system that bridges algorithmic semantics with I/O efficiency to accelerate the Re-Prefill phase. We first introduce \textit{ContiguousChunk}, a unified data management granularity that aligns KV cache pruning with I/O operations. All the mechanisms critical for I/O performance are performed at the granularity of ContiguousChunk, thereby eliminating read amplification. By exploiting the high similarity in important ContiguousChunk indices across layers, we propose intra- and inter-period asynchronous prefetching to break the sequential dependency between I/O and compute, effectively eliminating idle bubbles. Finally, we propose attention-guided cache management to retain semantically critical prefix data in memory. Evaluations on Qwen2.5 series models show that ContiguousKV achieves a 3.85x speedup in the Re-Prefill phase over the state-of-the-art offloading system IMPRESS, while maintaining high output quality.

Figures (16)

• Figure 1: The standard architecture of modern LLMs and the inference phases (i.e., prefill phase and decoding phase).
• Figure 2: Comparison between multi-head attention and grouped-query attention.
• Figure 3: Paradigm of offloading-based inference systems.
• Figure 4: Read amplification of existing SOTA inference system (e.g., IMPRESS 25fast-impress) when serving Qwen2.5-7B on the RTE dataset. The system adopts KV cache pruning methods to identify important KV caches, and the KV cache budget ratio is 25%.
• Figure 5: Latency breakdown of existing SOTA inference systems (IMPRESS 25fast-impress) among different KV cache budget ratios.

Theorems & Definitions (3)

• Definition 2.1
• Definition 4.1
• Definition 4.2