Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

EWSJF: An Adaptive Scheduler with Hybrid Partitioning for Mixed-Workload LLM Inference

Bronislav Sidik, Chaya Levi, Joseph Kampeas

TL;DR

This paper tackles the bottleneck of serving mixed-workload LLM inferences by introducing EWSJF, an adaptive upstream scheduler that learns workload structure in real time. EWSJF combines Refine-and-Prune to partition requests into performance-homogeneous queues, a Density-Weighted Scoring function to balance urgency and fairness, and a Bayesian meta-optimizer to continually tune parameters, all implemented as a pluggable module in vLLM. The approach yields substantial improvements in end-to-end throughput (over 30%) and dramatic reductions in Time-To-First-Token for short queries (up to 4x) across realistic, mixed workloads, while maintaining fairness. This work provides a practical, learning-based layer for efficient LLM serving and points to future directions in integrating adaptive scheduling with execution-level optimizations and multi-model environments.

Abstract

Serving Large Language Models (LLMs) under mixed workloads--short, latency-sensitive interactive queries alongside long, throughput-oriented batch requests--poses a fundamental scheduling challenge. Standard First-Come, First-Served (FCFS) policies suffer from severe head-of-line blocking, leading to high tail latency and underutilized hardware. We introduce EWSJF (Effective Workload-based Shortest Job First), an adaptive request-level scheduler that learns workload structure in real time to jointly improve fairness and throughput. EWSJF operates upstream of execution-level schedulers and integrates four components: (1) Refine-and-Prune, an unsupervised partitioning algorithm that discovers performance-homogeneous request groups; (2) Dynamic Queue Routing for assigning requests to these groups; (3) Density-Weighted Scoring, a context-aware prioritization function balancing urgency and fairness; and (4) Bayesian Meta-Optimization, which continuously tunes scoring and partitioning parameters based on live performance feedback. Implemented in vLLM, EWSJF improves end-to-end throughput by over 30% and reduces average Time-To-First-Token for short requests by up to 4x compared to FCFS. These results demonstrate that adaptive, learning-based request scheduling is a critical missing layer for efficient and responsive LLM serving. Implementation available at https://anonymous.4open.science/r/vllm_0110-32D8.

EWSJF: An Adaptive Scheduler with Hybrid Partitioning for Mixed-Workload LLM Inference

TL;DR

This paper tackles the bottleneck of serving mixed-workload LLM inferences by introducing EWSJF, an adaptive upstream scheduler that learns workload structure in real time. EWSJF combines Refine-and-Prune to partition requests into performance-homogeneous queues, a Density-Weighted Scoring function to balance urgency and fairness, and a Bayesian meta-optimizer to continually tune parameters, all implemented as a pluggable module in vLLM. The approach yields substantial improvements in end-to-end throughput (over 30%) and dramatic reductions in Time-To-First-Token for short queries (up to 4x) across realistic, mixed workloads, while maintaining fairness. This work provides a practical, learning-based layer for efficient LLM serving and points to future directions in integrating adaptive scheduling with execution-level optimizations and multi-model environments.

Abstract

Serving Large Language Models (LLMs) under mixed workloads--short, latency-sensitive interactive queries alongside long, throughput-oriented batch requests--poses a fundamental scheduling challenge. Standard First-Come, First-Served (FCFS) policies suffer from severe head-of-line blocking, leading to high tail latency and underutilized hardware. We introduce EWSJF (Effective Workload-based Shortest Job First), an adaptive request-level scheduler that learns workload structure in real time to jointly improve fairness and throughput. EWSJF operates upstream of execution-level schedulers and integrates four components: (1) Refine-and-Prune, an unsupervised partitioning algorithm that discovers performance-homogeneous request groups; (2) Dynamic Queue Routing for assigning requests to these groups; (3) Density-Weighted Scoring, a context-aware prioritization function balancing urgency and fairness; and (4) Bayesian Meta-Optimization, which continuously tunes scoring and partitioning parameters based on live performance feedback. Implemented in vLLM, EWSJF improves end-to-end throughput by over 30% and reduces average Time-To-First-Token for short requests by up to 4x compared to FCFS. These results demonstrate that adaptive, learning-based request scheduling is a critical missing layer for efficient and responsive LLM serving. Implementation available at https://anonymous.4open.science/r/vllm_0110-32D8.
Paper Structure (45 sections, 2 theorems, 13 equations, 7 figures, 11 tables, 2 algorithms)

This paper contains 45 sections, 2 theorems, 13 equations, 7 figures, 11 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLM Scheduling Policies
  4. Throughput-Optimal Schedulers.
  5. Fairness-Driven Schedulers.
  6. Agent and Multi-Step Schedulers.
  7. Execution and Scheduling Co-Design.
  8. Community Proposals.
  9. Workload-Aware Clustering and Partitioning
  10. Predictive Cost Modeling for LLM Scheduling
  11. EWSJF System Architecture
  12. Strategic Loop: Policy Generation
  13. Tactical Loop: Policy Execution
  14. Core Algorithmic Contributions
  15. Learnable and Context-Aware Scoring
  16. ...and 30 more sections

Key Result

theorem 1

For a system with $N$ historical requests and $k$ active queues, EWSJF guarantees $O(k)$ scheduling latency, non-blocking background optimization, and starvation freedom. (See Appendix app:theory for formal analysis and proofs).

Figures (7)

  • Figure 1: EWSJF architecture showing the interaction between tactical components (Dispatcher, Score Updater, Batch Builder) and strategic components (Monitor, Optimizer). The strategic loop supports both offline (historical) and online (real-time) modes.
  • Figure 2: Simulation of EWSJF's context-aware scoring. As the meta-optimizer adjusts scoring parameters over time, the relative priority of short, medium, and long queues shifts dynamically.
  • Figure 3: EWSJF speedup over FCFS across workload sizes and input rates. The x-axis shows input rate (log scale), and the y-axis shows percentage speedup in token throughput.
  • Figure 4: EWSJF throughput vs. queue count. Finer partitioning improves throughput, with diminishing returns beyond 20–30 queues.
  • Figure 5: Convergence behavior of the Bayesian meta-optimizer. The reward stabilizes after 5--8 trials.
  • ...and 2 more figures

Theorems & Definitions (3)

  • theorem 1: Efficiency and Stability
  • theorem 2: Starvation Freedom
  • proof