Slice-Level Scheduling for High Throughput and Load Balanced LLM Serving

TL;DR

This work tackles the challenge of scheduling LLM requests under unpredictable generation lengths and memory usage. It introduces Slice-Level Scheduling (SCLS), which partitions the maximal generation length into fixed slices and uses a dynamic-programming batcher plus a max-min offloader to optimize throughput and balance load across multiple LLM instances. Through real-data analysis and production-style experiments, SCLS achieves substantial gains (up to 315.8% throughput improvements) and significantly reduces average and tail latency while mitigating workload imbalance. The approach relies on accurate online serving-time and memory estimations, enabling adaptive batching and interval updates that respond to changing demand. Generation slicing is also positioned as a foundation for extending to continuous batching with further optimizations like amortized prefill and cache swapping, potentially broadening its practical impact on large-scale LLM serving systems.

Abstract

Large language models (LLMs) iteratively generate text token by token, with memory usage increasing with the length of generated token sequences. Since the request generation length is generally unpredictable, it is difficult to estimate the time and memory required to process requests, thus posing a challenge for effective request scheduling. Conventional sequence-level scheduling (SLS) serves requests in a first-come first-served (FCFS) manner with static batching where requests with short generation lengths are delayed until those with long ones have finished generation. Besides, to avoid out-of-memory (OOM) errors, SLS batches requests using a small batch size, which limits throughput. Recently proposed iteration-level scheduling (ILS) improves this with continuous batching, timely completing requests and dynamically adding new ones, but often limits the number of parallel-processing requests to OOM errors, thus compromising throughput. Moreover, both SLS and ILS fail to effectively balance workload across multiple LLM instances. To tackle these challenges, we propose slice-level scheduling (SCLS). By splitting the predefined maximal generation length limit into slices and serving batches slice by slice, it provides a precise range of serving time and memory usage for batched requests, laying the foundation for effective scheduling. Experiments confirm that compared with SLS and ILS schedulers, SCLS can improve throughput by up to 315.8% and greatly mitigate load imbalance with proposed batching and offloading algorithms.

Figures (19)

• Figure 1: Illustration of various scheduling techniques, where $r_i$ represents the $i$th arrived request.
• Figure 2: Inefficiency and load imbalance of SLS and ILS schedulers. Sub-figures (a) and (e) show that SCLS can significantly outperform SLS and ILS schedulers in terms of throughput and load balancing. Sub-figures (b) (d) present that SCLS's superior performance comes from the increase of batch size and decrease of the number of pad tokens and invalid tokens.
• Figure 3: Probability density function (PDF) and cumulative distribution function (CDF) of the request generation length in actual scenarios.
• Figure 4: SCLS system overview. The arrows represent data transferred between modules, where ❶ is requests, ❷ is temporary batches, ❸ and ❼ are estimated serving time, ❹ is estimated memory consumption, ❺, ❻, and ❽ are batches, ❾ is uncompleted requests, and ❿ is completed requests.
• Figure 5: Prefill latency under deepspeed-inference (DS).