Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Adaptive Rescheduling in Prefill-Decode Disaggregated LLM Inference

Zhibin Wang, Zetao Hong, Xue Li, Zibo Wang, Shipeng Li, Qingkai Meng, Qing Wang, Chengying Huan, Rong Gu, Sheng Zhong, Chen Tian

TL;DR

ARES addresses decode-phase workload imbalance in PD-disaggregated LLM inference by coupling a lightweight, LLM-native length predictor with a multi-stage decode rescheduler. The predictor leverages the last-layer hidden state of the final transformer layer to estimate remaining generation length, updated iteratively as decoding proceeds, achieving substantially lower MAE and overhead than auxiliary models. The rescheduler uses predicted workloads to migrate requests between decode instances, minimizing the variance of token loads over a horizon and overlapping KV-cache transfers to reduce disruption. Across real-system and large-scale simulations, ARES delivers up to 2.24x higher goodput and up to 74.77% lower tail latency (P99 TPOT), while preventing OOM events, demonstrating practical benefits for scalable LLM serving under long-output reasoning tasks.

Abstract

Large Language Model (LLM) inference has emerged as a fundamental paradigm. In real-world scenarios, variations in output length cause severe workload imbalance in the decode phase, particularly for long-output reasoning tasks. Existing systems, such as PD disaggregation architectures, rely on static prefill-to-decode scheduling, which often results in SLO violations and OOM failures under evolving decode workloads. In this paper, we propose ARES, an adaptive decoding rescheduling system powered by length prediction to anticipate future workloads. Our core contributions include: (1) A lightweight and continuous LLM-native prediction method that leverages LLM hidden state to model remaining generation length with high precision (reducing MAE by 49.42%) and low overhead (cutting predictor parameters by 93.28%); (2) A rescheduling solution in decode phase with : A dynamic balancing mechanism that integrates current and predicted workloads, reducing P99 TPOT by 74.77% and achieving up to 2.24 times higher goodput.

Adaptive Rescheduling in Prefill-Decode Disaggregated LLM Inference

TL;DR

ARES addresses decode-phase workload imbalance in PD-disaggregated LLM inference by coupling a lightweight, LLM-native length predictor with a multi-stage decode rescheduler. The predictor leverages the last-layer hidden state of the final transformer layer to estimate remaining generation length, updated iteratively as decoding proceeds, achieving substantially lower MAE and overhead than auxiliary models. The rescheduler uses predicted workloads to migrate requests between decode instances, minimizing the variance of token loads over a horizon and overlapping KV-cache transfers to reduce disruption. Across real-system and large-scale simulations, ARES delivers up to 2.24x higher goodput and up to 74.77% lower tail latency (P99 TPOT), while preventing OOM events, demonstrating practical benefits for scalable LLM serving under long-output reasoning tasks.

Abstract

Large Language Model (LLM) inference has emerged as a fundamental paradigm. In real-world scenarios, variations in output length cause severe workload imbalance in the decode phase, particularly for long-output reasoning tasks. Existing systems, such as PD disaggregation architectures, rely on static prefill-to-decode scheduling, which often results in SLO violations and OOM failures under evolving decode workloads. In this paper, we propose ARES, an adaptive decoding rescheduling system powered by length prediction to anticipate future workloads. Our core contributions include: (1) A lightweight and continuous LLM-native prediction method that leverages LLM hidden state to model remaining generation length with high precision (reducing MAE by 49.42%) and low overhead (cutting predictor parameters by 93.28%); (2) A rescheduling solution in decode phase with : A dynamic balancing mechanism that integrates current and predicted workloads, reducing P99 TPOT by 74.77% and achieving up to 2.24 times higher goodput.
Paper Structure (23 sections, 4 equations, 13 figures, 2 tables, 1 algorithm)

This paper contains 23 sections, 4 equations, 13 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. Prefill and Decode Phases
  4. Imbalance Workload in Decode Phase
  5. Generation Length Prediction
  6. Challenges
  7. Overview
  8. Generation Length Prediction
  9. Overview of Predictor
  10. Lightweight Target-LLM-Native Prediction
  11. Enhancing Precision with Continuous Prediction
  12. Implementation and Empirical Evaluation
  13. Decode Rescheduling
  14. From Prefill-to-Decode Scheduling to Decode-to-Decode Rescheduling
  15. Migration Algorithm
  16. ...and 8 more sections

Figures (13)

  • Figure 1: Different output lengths lead to significant load variations across decode instances.
  • Figure 2: Output length distribution.
  • Figure 3: Per-step execution time (TPOT) across three decode instances under PD disaggregation (1 prefill + 3 decode instances).
  • Figure 4: Current prediction methods.
  • Figure 5: System overview.
  • ...and 8 more figures