KVDirect: Distributed Disaggregated LLM Inference

TL;DR

KVDirect tackles the inefficiency of distributing disaggregated LLM inference across multiple nodes by introducing a tensor-centric, RDMA-based KV cache transfer mechanism and a pull-mode strategy. The system uses three operations—Connect(), Transfer(), and Complete()—to establish inter-node KV movement with metadata including Address, Dims, Shape, and Stride, enabling efficient, dynamic inter-node scheduling. Empirical results show up to a 55% reduction in per-request latency (and significant TBT improvements) over baselines under similar resource constraints, with bandwidths around 22.23 GB/s and clear advantages from pull-mode and block coalescing. KVDirect also demonstrates favorable scalability through configurable prefill/decode workers and is released as open-source, offering a practical path to higher throughput for disaggregated LLM inference in data-center environments.

Abstract

Large Language Models (LLMs) have become the new foundation for many applications, reshaping human society like a storm. Disaggregated inference, which separates prefill and decode stages, is a promising approach to improving hardware utilization and service quality. However, due to inefficient inter-node communication, existing systems restrict disaggregated inference to a single node, limiting resource allocation flexibility and reducing service capacity. This paper introduces KVDirect, which optimizes KV cache transfer to enable a distributed disaggregated LLM inference. KVDirect achieves this through the following contributions. First, we propose a novel tensor-centric communication mechanism that reduces the synchronization overhead in traditional distributed GPU systems. Second, we design a custom communication library to support dynamic GPU resource scheduling and efficient KV cache transfer. Third, we introduce a pull-based KV cache transfer strategy that reduces GPU resource idling and improves latency. Finally, we implement KVDirect as an open-source LLM inference framework. Our evaluation demonstrates that KVDirect reduces per-request latency by 55% compared to the baseline across diverse workloads under the same resource constraints.

Figures (17)

• Figure 1: The workflow of disaggregated LLM inference with an emphasis on KV cache.
• Figure 2: An example with three LLM inference scheduling approaches according to Splitwise patel2024splitwise. Here, we assume prompts A, B, and C arrive in a serial order.
• Figure 3: The message-based KV cache transfer with 4KB block size, where the blue and red arrows represent the communication over PCIe and network, respectively.
• Figure 4: The achieved bandwidth of UCX message-sending.
• Figure 5: An example of KVDirect's tensor communication.