RAPID-Serve: Resource-efficient and Accelerated P/D Intra-GPU Disaggregation

TL;DR

RAPID-Serve tackles the tension between latency SLOs and high throughput in LLM inference by overlapping prefill and decode concurrently on the same GPUs, avoiding inter-node KV transfers. It combines intra-GPU P/D concurrency with CU masking-based adaptive resource management to allocate compute units where needed while overlapping kernels. The approach analyzes overheads of hybrid batching and disaggregated serving, introduces a profiling-driven configuration strategy, and demonstrates substantial throughput and goodput gains under realistic SLO constraints. On AMD Instinct GPUs, RAPID-Serve achieves up to 4.1x unconstrained throughput and up to 32x goodput improvements over baselines, with particularly strong performance in resource-constrained environments.

Abstract

Two widely adopted techniques for LLM inference serving systems today are hybrid batching and disaggregated serving. A hybrid batch combines prefill and decode tokens of different requests in the same batch to improve resource utilization and throughput at the cost of increased latency per token. In contrast, disaggregated serving decouples compute-bound prefill and bandwidth-bound decode phases to optimize for service level objectives (SLOs) at the cost of resource under-utilization and KV-cache transfer overheads. To address the limitations of these techniques, we propose RAPID-Serve: a technique to concurrently execute prefill and decode on the same GPU(s) to meet latency SLOs while maintaining high throughput and efficient resource utilization. Furthermore, we propose Adaptive Resource Management for runtime compute resource allocation, optionally leveraging CU masking (a fine-grained Compute Unit partitioning feature on AMD Instinct\textsuperscript{TM} GPUs). RAPID-Serve provides up to 4.1x (average 1.7x) unconstrained throughput improvement and 32x and higher (average 4.9x) throughput improvement under SLO constraints, showing it as an effective strategy compared to the state-of-the-art approaches, particularly in resource-constrained environments.

Figures (12)

• Figure 1: Scheduling approaches. Consider a scenario where request A is in decode phase when B arrives; $B_p$ denotes prefill of request B and $A_d$ denotes decode of request A and , and $B_{p1}$ represents the first chunk of B’s prefill when chunked prefill is applied.
• Figure 2: Different Serving Techniques
• Figure 3: Resource requirements for inference phases. Y-axis represents performance degradation normalized to performance at 100% compute units (lower is better, 1=peak performance).
• Figure 4: Flow of a single request
• Figure 5: Main components of RAPID-Serve