Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Locality-aware Fair Scheduling in LLM Serving

Shiyi Cao, Yichuan Wang, Ziming Mao, Pin-Lun Hsu, Liangsheng Yin, Tian Xia, Dacheng Li, Shu Liu, Yineng Zhang, Yang Zhou, Ying Sheng, Joseph Gonzalez, Ion Stoica

TL;DR

The paper tackles the dual challenge of preserving prefix locality and ensuring fairness in multi-client LLM serving. It introduces Deficit Longest Prefix Match (DLPM), a deficit-based extension of Longest Prefix Match that maintains locality while providing approximate max-min fairness, and a distributed variant, Double Deficit LPM (D$^2$LPM), that balances locality, fairness, and load-balancing across multiple GPUs. The authors establish theoretical fairness bounds for DLPM and D$^2$LPM, implement these schedulers on a real LLM serving stack, and evaluate them across synthetic and real-world workloads. Results show up to $2.87\times$ higher throughput than VTC and up to $7.18\times$ lower latency for well-behaved clients compared with locality-aware baselines, demonstrating practical gains in both single- and multi-GPU settings. The work provides a principled framework for navigating the locality-fairness trade-off in online LLM inference, with tangible impact on system efficiency and predictable performance in multi-tenant deployments.

Abstract

Large language model (LLM) inference workload dominates a wide variety of modern AI applications, ranging from multi-turn conversation to document analysis. Balancing fairness and efficiency is critical for managing diverse client workloads with varying prefix patterns. Unfortunately, existing fair scheduling algorithms for LLM serving, such as Virtual Token Counter (VTC), fail to take prefix locality into consideration and thus suffer from poor performance. On the other hand, locality-aware scheduling algorithms in existing LLM serving frameworks tend to maximize the prefix cache hit rate without considering fair sharing among clients. This paper introduces the first locality-aware fair scheduling algorithm, Deficit Longest Prefix Match (DLPM), which can maintain a high degree of prefix locality with a fairness guarantee. We also introduce a novel algorithm, Double Deficit LPM (D$^2$LPM), extending DLPM for the distributed setup that can find a balance point among fairness, locality, and load-balancing. Our extensive evaluation demonstrates the superior performance of DLPM and D$^2$LPM in ensuring fairness while maintaining high throughput (up to 2.87$\times$ higher than VTC) and low per-client (up to 7.18$\times$ lower than state-of-the-art distributed LLM serving system) latency.

Locality-aware Fair Scheduling in LLM Serving

TL;DR

The paper tackles the dual challenge of preserving prefix locality and ensuring fairness in multi-client LLM serving. It introduces Deficit Longest Prefix Match (DLPM), a deficit-based extension of Longest Prefix Match that maintains locality while providing approximate max-min fairness, and a distributed variant, Double Deficit LPM (DLPM), that balances locality, fairness, and load-balancing across multiple GPUs. The authors establish theoretical fairness bounds for DLPM and DLPM, implement these schedulers on a real LLM serving stack, and evaluate them across synthetic and real-world workloads. Results show up to higher throughput than VTC and up to lower latency for well-behaved clients compared with locality-aware baselines, demonstrating practical gains in both single- and multi-GPU settings. The work provides a principled framework for navigating the locality-fairness trade-off in online LLM inference, with tangible impact on system efficiency and predictable performance in multi-tenant deployments.

Abstract

Large language model (LLM) inference workload dominates a wide variety of modern AI applications, ranging from multi-turn conversation to document analysis. Balancing fairness and efficiency is critical for managing diverse client workloads with varying prefix patterns. Unfortunately, existing fair scheduling algorithms for LLM serving, such as Virtual Token Counter (VTC), fail to take prefix locality into consideration and thus suffer from poor performance. On the other hand, locality-aware scheduling algorithms in existing LLM serving frameworks tend to maximize the prefix cache hit rate without considering fair sharing among clients. This paper introduces the first locality-aware fair scheduling algorithm, Deficit Longest Prefix Match (DLPM), which can maintain a high degree of prefix locality with a fairness guarantee. We also introduce a novel algorithm, Double Deficit LPM (DLPM), extending DLPM for the distributed setup that can find a balance point among fairness, locality, and load-balancing. Our extensive evaluation demonstrates the superior performance of DLPM and DLPM in ensuring fairness while maintaining high throughput (up to 2.87 higher than VTC) and low per-client (up to 7.18 lower than state-of-the-art distributed LLM serving system) latency.
Paper Structure (53 sections, 12 theorems, 8 equations, 14 figures, 3 tables, 2 algorithms)

This paper contains 53 sections, 12 theorems, 8 equations, 14 figures, 3 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background and Motivation
  3. Transformer-Based LLM Inference
  4. LLM Inference
  5. Prefix Caching and Locality
  6. LLM Serving Fairness
  7. The Trade-offs
  8. Locality vs. Fairness
  9. Locality vs. Load-Balancing
  10. Design Goals
  11. Preliminaries
  12. Fairness Properties
  13. Measurement of Service
  14. Deficit Longest Prefix Match (DLPM)
  15. Algorithm Design
  16. ...and 38 more sections

Key Result

Theorem 4.1

Under the DLPM scheme: for any time interval $[t_1,t_2)$, if two clients $f$ and $g$ are continuously backlogged. Then the difference in their received service are bounded: $\vert W_f(t_1,t_2) - W_g(t_1,t_2)\vert \leq 2 \cdot (U + Q^u)$, where $U=w_e\cdot L_{input} + w_q\cdot M$.

Figures (14)

  • Figure 1: DLPM achieves a new Pareto frontier considering locality and fairness in LLM serving. $Q$ is a hyper-parameter in DLPM, indicating how much we relax the fairness bound of DLPM. Results are obtained in a single A10 GPU.
  • Figure 2: LPM.
  • Figure 3: VTC.
  • Figure 4: DLPM (ours).
  • Figure 6: This paper addresses the conflict between fairness and locality through the DLPM mechanism (\ref{['sec:local_dlpm']}). It further addresses the conflict between locality and load balancing in distributed settings with the D$^2$LPM mechanism (\ref{['sec:dist_dlpm']}).
  • ...and 9 more figures

Theorems & Definitions (23)

  • Definition 3.1: Backlog
  • Theorem 4.1: Service bound between backlogged clients
  • proof
  • Theorem 4.2: Service bound between backlogged and non-backlogged clients
  • proof
  • Theorem 5.1: Service bound between backlogged clients
  • Theorem 5.2: Service bound between backlogged and non-backlogged clients
  • Theorem A.1: Service Bound
  • proof
  • Theorem A.2: Latency Bound
  • ...and 13 more