Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Large-Scale LLM Inference with Heterogeneous Workloads: Prefill-Decode Contention and Asymptotically Optimal Control

Ruihan Lin, Zezhen Ding, Zean Han, Jiheng Zhang

TL;DR

This work develops a scalable stochastic-control framework for large-scale LLM inference over heterogeneous workloads by modeling the system as a multiclass, many-server queue with two GPU modes: mixed (one prefill plus decodes) and solo (decode-only). A data-driven iteration-time law yields state-dependent service rates, and a fluid limit analysis reduces scheduling to a steady-state linear program that prescribes per-class prefill occupancy and decode routing. The authors design a gate-and-route control framework that implements the LP in practice and prove asymptotic optimality in the many-GPU limit under both bundled (completion-based revenue) and separate (phase-based revenue) pricing, with SLIs like latency and fairness incorporated. Numerical experiments calibrated on production hardware demonstrate revenue and queue convergence toward fluid optima and show that SLI-aware policies can enforce targeted performance while outperforming standard heuristics. The results offer actionable guidance for pricing, resource partitioning, and SLI-aware scheduling in production LLM inference platforms. We highlight a decode-buffer elimination property, a key structural result, and provide insights into the practical trade-offs between fairness, latency, and revenue in large GPU deployments.

Abstract

Large Language Models (LLMs) are rapidly becoming critical infrastructure for enterprise applications, driving unprecedented demand for GPU-based inference services. A key operational challenge arises from the two-phase nature of LLM inference: a compute-intensive \emph{prefill} phase that processes user input, followed by a memory-bound \emph{decode} phase that generates output tokens. When these phases share GPU resources, prefill tasks throttle the processing speed of concurrent decodes, creating state-dependent contention. This contention is further complicated by workload heterogeneity, as different applications exhibit vastly different input and output lengths. We develop a stochastic control framework for scheduling heterogeneous LLM workloads across large GPU clusters. We formulate LLM inference as a multiclass many-server queueing network with state-dependent service rates, grounded in empirical iteration-time measurements. We analyze the fluid approximation of this system and solve steady-state linear programs that characterize optimal resource allocation. We design gate-and-route policies that regulate prefill admission and decode routing, and prove that they are asymptotically optimal in the many-GPU limit under both bundled and separate token-pricing schemes. We further extend the framework to incorporate Service Level Indicators (SLIs) such as latency and fairness, providing a general approach to constrained scheduling. Numerical experiments calibrated to empirical iteration-time data demonstrate that our policies outperform standard serving heuristics.

Large-Scale LLM Inference with Heterogeneous Workloads: Prefill-Decode Contention and Asymptotically Optimal Control

TL;DR

This work develops a scalable stochastic-control framework for large-scale LLM inference over heterogeneous workloads by modeling the system as a multiclass, many-server queue with two GPU modes: mixed (one prefill plus decodes) and solo (decode-only). A data-driven iteration-time law yields state-dependent service rates, and a fluid limit analysis reduces scheduling to a steady-state linear program that prescribes per-class prefill occupancy and decode routing. The authors design a gate-and-route control framework that implements the LP in practice and prove asymptotic optimality in the many-GPU limit under both bundled (completion-based revenue) and separate (phase-based revenue) pricing, with SLIs like latency and fairness incorporated. Numerical experiments calibrated on production hardware demonstrate revenue and queue convergence toward fluid optima and show that SLI-aware policies can enforce targeted performance while outperforming standard heuristics. The results offer actionable guidance for pricing, resource partitioning, and SLI-aware scheduling in production LLM inference platforms. We highlight a decode-buffer elimination property, a key structural result, and provide insights into the practical trade-offs between fairness, latency, and revenue in large GPU deployments.

Abstract

Large Language Models (LLMs) are rapidly becoming critical infrastructure for enterprise applications, driving unprecedented demand for GPU-based inference services. A key operational challenge arises from the two-phase nature of LLM inference: a compute-intensive \emph{prefill} phase that processes user input, followed by a memory-bound \emph{decode} phase that generates output tokens. When these phases share GPU resources, prefill tasks throttle the processing speed of concurrent decodes, creating state-dependent contention. This contention is further complicated by workload heterogeneity, as different applications exhibit vastly different input and output lengths. We develop a stochastic control framework for scheduling heterogeneous LLM workloads across large GPU clusters. We formulate LLM inference as a multiclass many-server queueing network with state-dependent service rates, grounded in empirical iteration-time measurements. We analyze the fluid approximation of this system and solve steady-state linear programs that characterize optimal resource allocation. We design gate-and-route policies that regulate prefill admission and decode routing, and prove that they are asymptotically optimal in the many-GPU limit under both bundled and separate token-pricing schemes. We further extend the framework to incorporate Service Level Indicators (SLIs) such as latency and fairness, providing a general approach to constrained scheduling. Numerical experiments calibrated to empirical iteration-time data demonstrate that our policies outperform standard serving heuristics.
Paper Structure (63 sections, 16 theorems, 182 equations, 12 figures, 3 tables)

This paper contains 63 sections, 16 theorems, 182 equations, 12 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Prefill--decode contention.
  3. Workload heterogeneity.
  4. Our approach.
  5. Our contributions.
  6. Literature Review
  7. Organization
  8. Problem Formulation
  9. Preliminaries: Modern LLM Inference Systems
  10. Two stages of LLM inference.
  11. Continuous batching.
  12. Chunked prefill and GPU modes.
  13. Token Processing on GPUs
  14. Iteration time.
  15. The Stochastic Model
  16. ...and 48 more sections

Key Result

Theorem 1

Fix any finite horizon $T>0$. The sequence of fluid-scaled stochastic processes is tight in $\mathbb{D}([0,T],\mathbb{R}^d)$ under the Skorokhod $J_1$ topology, where $d$ is the total dimension of the vector above. Moreover, any subsequential weak limit $\bar{\mathcal{X}}(t)$ is almost surely continuous and, on $[0,T]$, satisfies the fluid model equations eq:fluid-qp–eq:fluid-y

Figures (12)

  • Figure 1: Schematic of the Dynamic GPU Scheduling Architecture. The system manages a cluster of $n$ GPUs with batch size $B$. GPUs transition between the Solo State (decode-only) and Mixed State (one prefill + decodes) based on assignments from the Prefill Scheduler. Completed prefills enter a virtual Decode Buffer, from which the Decode Scheduler populates available slots in either state.
  • Figure 2: Comparison of Revenue and Queue Lengths under Bundled vs. Separate Charging Schemes (C0: class 0, C1: class 1).
  • Figure 3: Calibration of mixed-iteration hyperparameters $(\alpha,\beta)$ for Qwen-8B (left) and Qwen-4B (right). Dots show the empirical mean iteration time $\tau$ for each chunk size $C$, and the solid line is the OLS fit $\tau = \alpha + \beta C$. For Qwen-8B, the fitted line is $\tau = 0.0174 + 6.2\times 10^{-5}\,C$ with $R^2 = 0.998$; for Qwen-4B, it is $\tau = 0.0152 + 3.6\times 10^{-5}\,C$ with $R^2 = 0.997$.
  • Figure 4: Asymptotic convergence of Revenue and Queue Lengths under the Gate-and-Route policy.
  • Figure 5: Occupancy convergence under the Gate-and-Route Policy. Top: Class 0 (Decode-Heavy). Bottom: Class 1 (Prefill-Heavy). Note the loose convergence for decode occupancy ($y$).
  • ...and 7 more figures

Theorems & Definitions (32)

  • Theorem 1: Fluid limit
  • Proposition 1: Decode-buffer elimination
  • Theorem 2: Asymptotic optimality of occupancy-based Gate-and-Route Policy
  • Theorem 3: Asymptotic optimality under separate charging
  • Theorem 4: Occupancy Convergence and Asymptotic Optimality of SLI-Aware Policy
  • proof
  • proof
  • Lemma EC.1
  • proof
  • Lemma EC.2
  • ...and 22 more