semi-PD: Towards Efficient LLM Serving via Phase-Wise Disaggregated Computation and Unified Storage

TL;DR

The paper tackles LLM serving latency under TTFT and TPOT constraints by combining phase-wise disaggregated computation with unified storage. It introduces semi-PD, featuring a computational resource controller based on MPS and a unified memory manager to coordinate prefill and decode workers, plus a low-overhead, SLO-aware dynamic partitioning algorithm that adapts to workload shifts. Empirical results show 1.27–2.58× lower average end-to-end latency and 1.55–1.72× more SLO-compliant requests on multiple model families, illustrating practical benefits for scalable LLM deployment. By preserving disaggregated computation while avoiding storage inefficiencies and KV cache transfers, semi-PD offers a cost-effective path to high-throughput, low-latency LLM serving in both single-node and cluster environments.

Abstract

Existing large language model (LLM) serving systems fall into two categories: 1) a unified system where prefill phase and decode phase are co-located on the same GPU, sharing the unified computational resource and storage, and 2) a disaggregated system where the two phases are disaggregated to different GPUs. The design of the disaggregated system addresses the latency interference and sophisticated scheduling issues in the unified system but leads to storage challenges including 1) replicated weights for both phases that prevent flexible deployment, 2) KV cache transfer overhead between the two phases, 3) storage imbalance that causes substantial wasted space of the GPU capacity, and 4) suboptimal resource adjustment arising from the difficulties in migrating KV cache. Such storage inefficiency delivers poor serving performance under high request rates. In this paper, we identify that the advantage of the disaggregated system lies in the disaggregated computation, i.e., partitioning the computational resource to enable the asynchronous computation of two phases. Thus, we propose a novel LLM serving system, semi-PD, characterized by disaggregated computation and unified storage. In semi-PD, we introduce a computation resource controller to achieve disaggregated computation at the streaming multi-processor (SM) level, and a unified memory manager to manage the asynchronous memory access from both phases. semi-PD has a low-overhead resource adjustment mechanism between the two phases, and a service-level objective (SLO) aware dynamic partitioning algorithm to optimize the SLO attainment. Compared to state-of-the-art systems, semi-PD maintains lower latency at higher request rates, reducing the average end-to-end latency per request by 1.27-2.58x on DeepSeek series models, and serves 1.55-1.72x more requests adhering to latency constraints on Llama series models.

Figures (16)

• Figure 1: Illustration of the pros and cons of the different computation and storage patterns. semi-PD can have the advantages of both disaggregated computation and unified storage.
• Figure 2: Comparison among different computation and storage design paradigms for LLM serving.
• Figure 3: Storage imbalance between prefill instance and decode instance. (1) and (2) are measured with Llama3-8B, on ShareGPT and LongBench datasets, respectively. We set TP=2/1 for prefill/decode instances. (3) and (4) are measured with Llama3-70B, on ShareGPT and LongBench datasets, respectively. We set TP=4 for both instances.
• Figure 4: The consistency between storage shortage and latency explosion. The data is measured on an A100 40GB GPU, and the reported request rate is per GPU. P90 TPOT increases drastically when the storage space is exhausted. For comparison, we draw the P90 TPOT curve under the ideal storage.
• Figure 5: System overview of semi-PD.