Efficient Multi-round LLM Inference over Disaggregated Serving

TL;DR

AMPD addresses efficient multi-round LLM inference under PD disaggregation by pairing online adaptive scheduling (adaptive routing and prefill reordering) with offline ILP-based deployment planning. The system uses a piecewise $\alpha$-$\beta$ performance model to estimate $T_{\text{pre}}$, $T_{\text{dec}}$, and $T_{\text{kv}}$ and a MILP solver to minimize the P95 latency bottleneck across deployed replicas, subject to GPU capacity. Empirical evaluation across multiple models and traces shows substantial improvements in SLO attainment over baselines, driven by balancing TTFT and ITL through adaptive routing and reordering. The work demonstrates that integrating PD disaggregation with adaptive control yields practical performance gains for complex, multi-round LLM workflows.

Abstract

With the rapid evolution of Large Language Models (LLMs), multi-round workflows, such as autonomous agents and iterative retrieval, have become increasingly prevalent. However, this raises hurdles for serving LLMs under prefill-decode (PD) disaggregation, a widely adopted paradigm that separates the compute-bound prefill phase and memory-bound decode phase onto individual resources. Specifically, existing systems overlook the interleaved prefill-decode workload pattern in multi-round inference, leading to sub-optimal handling of the incremental prefill workloads and model deployment for the two phases. In this work, we present AMPD, a brand new disaggregated serving framework for multi-round LLM inference. The core of AMPD is to coordinate the prefill workloads based on real-time workloads by adaptively determining where to carry out these workloads and how they are scheduled, in order to maximize service level objective (SLO) attainment. In addition, we tailor a planning algorithm for our scenario, facilitating the deduction of optimal resource allocation and parallel strategies for the two phases. Empirical results demonstrate that AMPD substantially improves SLO attainment compared to state-of-the-art baselines.

Figures (8)

• Figure 1: Illustration of multi-round LLM inference.
• Figure 2: System overview of AMPD.
• Figure 3: Illustration of disaggregated LLM serving where prefill and decode workers are with different parallelism configurations.
• Figure 4: End-to-end comparison. The top three rows are the SLO attainment under different configurations (traces and request arrival rates) for the three models. The bottom row is a detailed breakdown of Llama-3.1-70B with a request arrival rate of 2 reqs/s, including the average TTFT for initial prefill, average TTFT for incremental prefill, and average ITL for decoding.
• Figure 5: Ablation studies (Llama3.1-70B, 2 reqs/s). Left: DuReader. Middle: GAIA. Right: Proportion of local and remote execution.