Is the GPU Half-Empty or Half-Full? Practical Scheduling Techniques for LLMs

TL;DR

The paper investigates how to efficiently schedule concurrent requests for large language model inference, focusing on two architectural levels: engine-level schedulers on each server and load-balancing across replicas. By surveying existing policies from the literature and production systems, the authors show a trade-off between high performance and complexity, motivating two new drop-in techniques: LARRY (engine-level) and SAL (token-aware load balancing). In extensive experiments on production traces with both single and multi-server deployments, LARRY and SAL outperform baselines, reducing time-to-first-token and improving throughput while maintaining low preemption rates. The work demonstrates that memory-aware, token-aware scheduling can yield practical, deployable gains for real-world LLM serving deployments.

Abstract

Serving systems for Large Language Models (LLMs) improve throughput by processing several requests concurrently. However, multiplexing hardware resources between concurrent requests involves non-trivial scheduling decisions. Practical serving systems typically implement these decisions at two levels: First, a load balancer routes requests to different servers which each hold a replica of the LLM. Then, on each server, an engine-level scheduler decides when to run a request, or when to queue or preempt it. Improved scheduling policies may benefit a wide range of LLM deployments and can often be implemented as "drop-in replacements" to a system's current policy. In this work, we survey scheduling techniques from the literature and from practical serving systems. We find that schedulers from the literature often achieve good performance but introduce significant complexity. In contrast, schedulers in practical deployments often leave easy performance gains on the table but are easy to implement, deploy and configure. This finding motivates us to introduce two new scheduling techniques, which are both easy to implement, and outperform current techniques on production workload traces.

Figures (12)

• Figure 1: Typical serving architecture where request scheduling occurs at two levels.
• Figure 2: Throughput during the decode phase as a function of the batch size.
• Figure 3: Roofline analysis of SW-Chat requests for Llama-3 8B on an A100 GPU splitwise.
• Figure 4: 20-minute excerpt from a representative production request trace workload.
• Figure 5: Fraction of a request's memory consumption used by prompt splitwise.