ELIS: Efficient LLM Iterative Scheduling System with Response Length Predictor

TL;DR

ELIS tackles head-of-line blocking in LLM iterative batching by introducing Iterative Shortest Remaining Time First (ISRTF) scheduling guided by a response-length predictor built on BGE embeddings. The predictor updates predictions iteratively as more of the response is revealed, enabling model-agnostic prioritization and dynamic adaptation. Implemented as Kubernetes components with vLLM, ELIS scales to multi-node deployments and shows up to 19.6% reduction in average JCT and near-linear throughput (up to 18.77 RPS on 50 workers) on real traces. The work demonstrates practical, production-grade improvements for cloud-native LLM serving with a focus on latency, preemption (where applicable), and iterative scheduling.

Abstract

We propose ELIS, a serving system for Large Language Models (LLMs) featuring an Iterative Shortest Remaining Time First (ISRTF) scheduler designed to efficiently manage inference tasks with the shortest remaining tokens. Current LLM serving systems often employ a first-come-first-served scheduling strategy, which can lead to the "head-of-line blocking" problem. To overcome this limitation, it is necessary to predict LLM inference times and apply a shortest job first scheduling strategy. However, due to the auto-regressive nature of LLMs, predicting the inference latency is challenging. ELIS addresses this challenge by training a response length predictor for LLMs using the BGE model, an encoder-based state-of-the-art model. Additionally, we have devised the ISRTF scheduling strategy, an optimization of shortest remaining time first tailored to existing LLM iteration batching. To evaluate our work in an industrial setting, we simulate streams of requests based on our study of real-world user LLM serving trace records. Furthermore, we implemented ELIS as a cloud-native scheduler system on Kubernetes to evaluate its performance in production environments. Our experimental results demonstrate that ISRTF reduces the average job completion time by up to 19.6%.

Figures (7)

• Figure 1: BGE CLS vector distance with different groups.
• Figure 2: (a) Illustration of prediction procedure where each step (iteration) comprises of 50 tokens and (b) MAE of predictor for each step.
• Figure 3: Overall architecture of ELIS.
• Figure 4: Request interval distribution of LLM serving. The Gamma PDF and Poisson PMF distributions were fitted based on the observed data.
• Figure 5: (left)JCT comparison between FCFS and ISRTF where each experiment uses a multiple of average throughput. Bar represents the average value and each tick represents the minimum and the maximum value of each experiment. (right) Average JCT and queuing delay of lam13 with 5.0x RPS (case highlighted in gray shading).