Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Justitia: Fair and Efficient Scheduling for LLM Applications

Mingyan Yang, Guanjie Wang, Manqi Luo, Yifei Liu, Chen Chen, Han Zhao, Yu Feng, Quan Chen, Minyi Guo

TL;DR

Problem: scheduling multiple correlated LLM inferences (LLM applications) on shared GPUs while ensuring fast completion with worst-case guarantees. Approach: Justitia combines memory-centric KV-cache cost modeling with a KV-token-time metric $c = \sum_{i=1}^{d}(p+i) = pd + \frac{d^2}{2}$, app-specific MLP-based demand prediction, and virtual-time fair queuing using GPS to compute a fair completion order, enabling saturated sequential serving. Key results: a constant-delay bound $f_j - \bar{f}_j \leq 2 c_{max} + \dfrac{C_{max}}{M}$; substantial average JCT reductions (e.g., 57.5% vs VTC) and strong fairness (e.g., 92% of apps no later than under VTC). Significance: offers a practical, theoretically grounded framework for end-to-end LLM application scheduling that performs well under contention and avoids head-of-line blocking while preserving fairness.

Abstract

In the era of Large Language Models (LLMs), it has been popular to launch a series of LLM inferences -- we call an LLM application -- to better solve real-world problems. When serving those applications in shared GPU servers, the schedulers are expected to attain fast application completions with guaranteed worst-case performance. However, mainstream LLM schedulers fail to behave well for LLM applications -- due to head-of-line blocking or over-constrained resource allocation. In this paper, we propose to serve LLM applications in a fair and also efficient manner. To this end, we design Justitia, a novel scheduler with three key techniques. First, given that memory is prevalently a bottleneck for mainstream inference frameworks like vLLM, Justitia models the service cost of LLM applications in a memory-centric manner. Meanwhile, it uses a simple neural network model to conduct light-weight and also accurate demand prediction. Moreover, Justitia adopts a virtual-time based fair queuing algorithm to reduce the overall performance with guaranteed worst-case delay. We have implemented Justitia atop vLLM, and experimental results involving diverse LLM applications show that it can substantially enhance the scheduling efficiency with fairness preserved.

Justitia: Fair and Efficient Scheduling for LLM Applications

TL;DR

Problem: scheduling multiple correlated LLM inferences (LLM applications) on shared GPUs while ensuring fast completion with worst-case guarantees. Approach: Justitia combines memory-centric KV-cache cost modeling with a KV-token-time metric , app-specific MLP-based demand prediction, and virtual-time fair queuing using GPS to compute a fair completion order, enabling saturated sequential serving. Key results: a constant-delay bound ; substantial average JCT reductions (e.g., 57.5% vs VTC) and strong fairness (e.g., 92% of apps no later than under VTC). Significance: offers a practical, theoretically grounded framework for end-to-end LLM application scheduling that performs well under contention and avoids head-of-line blocking while preserving fairness.

Abstract

In the era of Large Language Models (LLMs), it has been popular to launch a series of LLM inferences -- we call an LLM application -- to better solve real-world problems. When serving those applications in shared GPU servers, the schedulers are expected to attain fast application completions with guaranteed worst-case performance. However, mainstream LLM schedulers fail to behave well for LLM applications -- due to head-of-line blocking or over-constrained resource allocation. In this paper, we propose to serve LLM applications in a fair and also efficient manner. To this end, we design Justitia, a novel scheduler with three key techniques. First, given that memory is prevalently a bottleneck for mainstream inference frameworks like vLLM, Justitia models the service cost of LLM applications in a memory-centric manner. Meanwhile, it uses a simple neural network model to conduct light-weight and also accurate demand prediction. Moreover, Justitia adopts a virtual-time based fair queuing algorithm to reduce the overall performance with guaranteed worst-case delay. We have implemented Justitia atop vLLM, and experimental results involving diverse LLM applications show that it can substantially enhance the scheduling efficiency with fairness preserved.
Paper Structure (20 sections, 1 theorem, 9 equations, 12 figures, 3 tables)

This paper contains 20 sections, 1 theorem, 9 equations, 12 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background
  3. LLM Application Serving: The Basics
  4. Related Works and Their Limitations
  5. Motivation
  6. Insight
  7. Challenges
  8. Solution
  9. Memory-Centric Cost Modeling
  10. MLP-based Demand Prediction
  11. Application-level Fair Queuing
  12. Delay Analysis
  13. Evaluation
  14. Setup
  15. End-to-end Scheduling Performance
  16. ...and 5 more sections

Key Result

Theorem 1

With Justitia, an application is guaranteed to complete within a constant time after its completion in GPS, i.e., for each app-$j$, we have

Figures (12)

  • Figure 1: Given two competing LLM applications, serving them sequentially in a saturated manner can reduce the average completion time with no per-application delay.
  • Figure 2: Examples of typical LLM applications.
  • Figure 3: KV block usage when running two DocMerging (DM) applications under different scheduling schemes.
  • Figure 5: Demand modeling of an LLM inference. $p$ and $d$ are respectively the prefill and decoding token length. The service cost is then depicted as the accumulative KV cache occupation (i.e., KV token-time): $pd +\frac{d^2}{2}$.
  • Figure 6: Predict the resource demand from the input prompt with MLP model.
  • ...and 7 more figures

Theorems & Definitions (2)

  • Theorem 1: Constant delay bound
  • proof