Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FairBatching: Fairness-Aware Batch Formation for LLM Inference

Hongtao Lyu, Boyue Liu, Mingyu Wu, Haibo Chen

TL;DR

FairBatching tackles the problem of balancing TTFT latency and TPOT throughput in LLM inference by introducing an SLO-aware, fairness-driven scheduler. It replaces decode-prioritizing batch formation with an adaptive time-budget batch capacity and a three-phase batch packing strategy that reallocates compute from decoding slack to absorb prefill bursts, complemented by a Prefill Admission Budget for upper-level load balancing. Empirical results on multiple models, traces, and hardware show significant tail latency reductions and substantial capacity gains in both single-node and clustered deployments. The work provides a practical, scalable approach to improving QoS in production LLM serving by ensuring fair resource sharing between prefill and decode tasks and enabling more effective global load management.

Abstract

Large language model (LLM) inference systems face a fundamental tension between minimizing Time-to-First-Token (TTFT) latency for new requests and maintaining a high, steady token generation rate (low Time-Per-Output-Token, or TPOT) for ongoing requests. Existing stall-free batching schedulers proposed by Sarathi, while effective at preventing decode stalls, introduce significant computational unfairness. They prioritize decode tasks excessively, simultaneously leading to underutilized decode slack and unnecessary prefill queuing delays, which collectively degrade the system's overall quality of service (QoS). This work identifies the root cause of this unfairness: the non-monotonic nature of Time-Between-Tokens (TBT) as a scheduling metric and the rigid decode-prioritizing policy that fails to adapt to dynamic workload bursts. We therefore propose FairBatching, a novel LLM inference scheduler that enforces fair resource allocation between prefill and decode tasks. It features an adaptive batch capacity determination mechanism, which dynamically adjusts the computational budget to improve the GPU utilization without triggering SLO violations. Its fair and dynamic batch formation algorithm breaks away from the decode-prioritizing paradigm, allowing computation resources to be reclaimed from bursting decode tasks to serve prefill surges, achieving global fairness. Furthermore, FairBatching provides a novel load estimation method, enabling more effective coordination with upper-level schedulers. Implemented and evaluated on realistic traces, FairBatching significantly reduces TTFT tail latency by up to 2.29x while robustly maintaining TPOT SLOs, achieving overall 20.0% improvement in single-node capacity and 54.3% improvement in cluster-level capacity.

FairBatching: Fairness-Aware Batch Formation for LLM Inference

TL;DR

FairBatching tackles the problem of balancing TTFT latency and TPOT throughput in LLM inference by introducing an SLO-aware, fairness-driven scheduler. It replaces decode-prioritizing batch formation with an adaptive time-budget batch capacity and a three-phase batch packing strategy that reallocates compute from decoding slack to absorb prefill bursts, complemented by a Prefill Admission Budget for upper-level load balancing. Empirical results on multiple models, traces, and hardware show significant tail latency reductions and substantial capacity gains in both single-node and clustered deployments. The work provides a practical, scalable approach to improving QoS in production LLM serving by ensuring fair resource sharing between prefill and decode tasks and enabling more effective global load management.

Abstract

Large language model (LLM) inference systems face a fundamental tension between minimizing Time-to-First-Token (TTFT) latency for new requests and maintaining a high, steady token generation rate (low Time-Per-Output-Token, or TPOT) for ongoing requests. Existing stall-free batching schedulers proposed by Sarathi, while effective at preventing decode stalls, introduce significant computational unfairness. They prioritize decode tasks excessively, simultaneously leading to underutilized decode slack and unnecessary prefill queuing delays, which collectively degrade the system's overall quality of service (QoS). This work identifies the root cause of this unfairness: the non-monotonic nature of Time-Between-Tokens (TBT) as a scheduling metric and the rigid decode-prioritizing policy that fails to adapt to dynamic workload bursts. We therefore propose FairBatching, a novel LLM inference scheduler that enforces fair resource allocation between prefill and decode tasks. It features an adaptive batch capacity determination mechanism, which dynamically adjusts the computational budget to improve the GPU utilization without triggering SLO violations. Its fair and dynamic batch formation algorithm breaks away from the decode-prioritizing paradigm, allowing computation resources to be reclaimed from bursting decode tasks to serve prefill surges, achieving global fairness. Furthermore, FairBatching provides a novel load estimation method, enabling more effective coordination with upper-level schedulers. Implemented and evaluated on realistic traces, FairBatching significantly reduces TTFT tail latency by up to 2.29x while robustly maintaining TPOT SLOs, achieving overall 20.0% improvement in single-node capacity and 54.3% improvement in cluster-level capacity.
Paper Structure (32 sections, 17 equations, 8 figures, 5 tables, 1 algorithm)

This paper contains 32 sections, 17 equations, 8 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Characterizing Unfairness in LLM Serving
  3. LLM Inference
  4. SLO Metrics for LLM Inference
  5. Existing LLM Serving Scheduling
  6. Unfairness in LLM Serving
  7. Design of FairBatching
  8. SLO Formulation in FairBatching
  9. Batch Capacity Determination
  10. Fair and Dynamic Batch Formation
  11. Integrated with Upper-Level schedulers
  12. Implementation
  13. Evaluation
  14. Experimental setup
  15. Single-Node Performance
  16. ...and 17 more sections

Figures (8)

  • Figure 1: Unfairness in stall-free batching: decode tasks accumulate slack during prefill idleness but still constrain prefill capacity during prefill bursts, causing TTFT violations.
  • Figure 2: Decisions made by inference systems. (a) Ideal, steady request arrival with fair resource sharing. (b) Realistic bursty arrivals create resource contention periods and idle periods. (c) vLLM's prefill-first policy causes decode starvation. (d) Sarathi's strict TBT guarantees causes TTFT violations. (e) FB allows prefill bursts without breaking TPOT guarantees, achieving the best overall service quality.
  • Figure 3: The prefill capacity is calculated as the remaining capacity after subtracting the computational resources already allocated to decode tasks. The prefill admission budget is the prefill capacity minus the already allocated prefill tasks.
  • Figure 4: Dataset requests distribution
  • Figure 5: Effective RPS of FairBatching under different loads, tested under different models and datasets
  • ...and 3 more figures