Shared Disk KV Cache Management for Efficient Multi-Instance Inference in RAG-Powered LLMs

TL;DR

The paper tackles growing TTFT and latency in RAG-powered LLM inference by introducing disk-based KV caching to prestore and share key-value states of document chunks. It presents RAG-DCache for single-instance setups and a multi-instance extension, Shared RAG-DCache, that proactively generates and shares KV caches across instances using a central manager and a background KV Cache Generator, leveraging query locality. Experimental evaluations show TTFT reductions of about $10$–$20\%$ and throughput gains around $14$–$15\%$ for single-instance LLMs, with Shared RAG-DCache delivering up to $71\%$ higher throughput and up to $65\%$ lower latency under heavier loads; CPU-based KV generation tends to yield the best performance in multi-instance settings. The results demonstrate practical improvements for scalable RAG-enabled inference on commodity hardware, highlighting the role of disk-resident caches and cross-instance sharing in reducing redundant prefill computations.

Abstract

Recent large language models (LLMs) face increasing inference latency as input context length and model size continue to grow. In particular, the retrieval-augmented generation (RAG) technique, which enhances LLM responses by incorporating external knowledge, exacerbates this issue by significantly increasing the number of input tokens. This expansion in token length leads to a substantial rise in computational overhead, particularly during the prefill stage, resulting in prolonged time-to-first-token (TTFT). To address this issue, this paper proposes a method to reduce TTFT by leveraging a disk-based key-value (KV) cache to lessen the computational burden during the prefill stage. We also introduce a disk-based shared KV cache management system, called Shared RAG-DCache, for multi-instance LLM RAG service environments. This system, together with an optimal system configuration, improves both throughput and latency under given resource constraints. Shared RAG-DCache exploits the locality of documents related to user queries in RAG, as well as the queueing delay in LLM inference services. It proactively generates and stores disk KV caches for query-related documents and shares them across multiple LLM instances to enhance inference performance. In experiments on a single host equipped with 2 GPUs and 1 CPU, Shared RAG-DCache achieved a 15~71% increase in throughput and up to a 12~65% reduction in latency, depending on the resource configuration.

Figures (11)

• Figure 1: CDF of query related documents.
• Figure 2: Inference Latency Based on Requests Per Second in a Multi-Instance LLM(with RAG) Service Environment.
• Figure 3: Vector DB creation process and tuple structure for RAG.
• Figure 4: Design and Operation of RAG-DCache.
• Figure 5: Shared RAG-DCache Architecture.