PLA-Serve: A Prefill-Length-Aware LLM Serving System

TL;DR

PLA-Serve tackles the intra-prefill interference caused by mixing long compute-bound and short memory-bound prefill requests in PD-based LLM serving. It introduces a two-pronged approach: (i) disaggregate prefill workloads by length, and (ii) apply CUDA Graph–based batching and an AWD scheduler to optimize short-prefill throughput, complemented by temporal and spatial disaggregation across prefill instances to balance load. The system demonstrates significant improvements in throughput and latency, and reduced SLO violations, outperforming state-of-the-art frameworks like SGLang and vLLM under PD disaggregation, especially under high concurrency and mixed workloads. The results suggest practical gains for real-world multi-turn LLM services, enabling more consistent latency and higher utilization across heterogeneous workloads.

Abstract

PLA-Serve identifies and disaggregates requests with different prompt lengths in LLM serving to reduce TTFT latency. While recent systems have decoupled the prefill and decode stages to improve throughput, they still rely on unified scheduling policies that fail to adapt to heterogeneous workload characteristics. We observe that prompt-length variations lead to distinct performance bottlenecks, motivating an adaptive scheduling strategy. PLA-Serve disaggregates multi-turn long-prefill requests from short-prefill ones and introduces a length-aware smart batching mechanism for short-prefill workloads. It adopts a dual-queue design that supports temporal disaggregation on a single prefill instance or spatial disaggregation across multiple instances. For short-prefill batches, a batch waiting window and CUDA Graph-based clustering mitigate interference from heterogeneous computation, reducing batching delay and lowering average latency. In real multi-turn workloads, PLA-Serve reduces prefill latency by over 30% compared to vanilla SGLang under prefill**--**decode disaggregation, and further decreases SLO violations by 28% in multi-instance deployments with vanilla data-parallel configuration. Compared to the SGLang router with load balancing, it further lowers SLO violations by 12% in multi-GPU settings. Under high concurrency and mixed-request scenarios, PLA-Serve improves request throughput by 35% serving Qwen2.5-32B model for prefill instance, demonstrating its effectiveness in optimizing heterogeneous LLM serving workloads.

Figures (8)

• Figure 1: P90 TTFT of long-prefill requests under varying concurrency levels for long and short requests. The long-prefill requests have more than 1K tokens, while the short ones have fewer than 64 tokens. We concurrently run them on a single H200 GPU and serve by Qwen2.5-32B qwen2025qwen25technicalreport. The dashed lines indicate the latency when only long-prefill requests are served.
• Figure 2: The token length distribution of multi-turn dialogues in the real LMsys-Chat-1M dataset. The left plot shows the prompt length in the first turn (including the system prompt by default), where approximately 63% of requests contain fewer than 256 tokens. In subsequent turns, the proportion of prompts shorter than 256 tokens increases to an average of 81%.
• Figure 3: P90 TTFT of short-prefill requests under varying concurrency levels for long and short requests. The dashed lines indicate the latency when only short-prefill requests are served. Other setups are the same as those in Figure \ref{['fig:interfere']}.
• Figure 4: Resource utilization during multi-turn LLM inference. Long-context requests saturate tensor cores during prefill (compute-bound), while short, frequent requests and re-prefill stages are memory-bound with high HBM usage—illustrating the interference between compute- and memory-bound workloads in shared serving systems.
• Figure 5: Average latency and throughput curves over varying waiting windows. The larger the waiting window, the more short-prefill requests will be batched. The serving system runs on an H200 GPU and a 14B model, with 64-way concurrency for short-prefill requests (prompt length less than 256 tokens).