SCORPIO: Serving the Right Requests at the Right Time for Heterogeneous SLOs in LLM Inference

TL;DR

SCORPIO addresses suboptimal SLO attainment in throughput-oriented LLM serving by introducing TTFT Guard, TPOT Guard, and a predictive module that exploit SLO heterogeneity. The system uses TRP-based crediting, VBS-based admission control, and least-deadline-first reordering to adaptively admit and batch requests. It demonstrates up to 14.4× better goodput and up to 46.5% higher SLO adherence than baselines on diverse workloads and traces. The objective is to maximize $Goodput(pi) = |R_{good}(pi)| / T$ and $Adherence(pi) = |R_{good}(pi)| / |R|$, enabling practical MaaS deployments with heterogeneous latency requirements.

Abstract

Existing Large Language Model (LLM) serving systems prioritize maximum throughput. They often neglect Service Level Objectives (SLOs) such as Time to First Token (TTFT) and Time Per Output Token (TPOT), which leads to suboptimal SLO attainment. This paper introduces SCORPIO, an SLO-oriented LLM serving system designed to maximize system goodput and SLO attainment for workloads with heterogeneous SLOs. Our core insight is to exploit SLO heterogeneity for adaptive scheduling across admission control, queue management, and batch selection. SCORPIO features a TTFT Guard, which employs least-deadline-first reordering and rejects unattainable requests, and a TPOT Guard, which utilizes a VBS-based admission control and a novel credit-based batching mechanism. Both guards are supported by a predictive module. Evaluations demonstrate that SCORPIO improves system goodput by up to 14.4X and SLO adherence by up to 46.5% compared to state-of-the-art baselines.

Figures (9)

• Figure 1: Comparison of throughput-oriented and SLO-oriented scheduling approaches. The throughput-oriented scheduler greedily admits and processes requests without considering per-request SLOs. During the prefill phase, request 4 violates its SLO constraint (4 steps> 3 steps). The SLO-oriented scheduler prevents such violations through least-deadline-first prioritization. During the decode phase, we assume a normalized decode step time as 0.25$\times$ BatchSize. The throughput-oriented scheduler batch all requests in each step (BatchSize=6), causing requests 0, 1, 3, and 5 to violate their TPOT constraints (each step consumes a time of 1.5). In contrast, the SLO-oriented scheduler rejects unattainable requests (e.g., request 5) and implements an adaptive fine-grained batching strategy (BatchSize=4). This strategy allows requests with looser TPOT SLOs (requests 2 and 4) to skip certain iterations, ensuring all admitted requests satisfy their TPOT constraints.
• Figure 2: Impact of input sequence length (a), batch size (b), and output token index (c) on prefill latency and inter-token latency (ITL).
• Figure 3: The architecture of Scorpio.
• Figure 4: Impact of different scheduling strategies on goodput and SLO adherence vs QPS.
• Figure 5: Cumulative number of SLO-met requests over time on the LMSYS and ShareGPT dataset.

Theorems & Definitions (1)

• Definition 1: TPOT-relative Proportionality (TRP)