TokenScale: Timely and Accurate Autoscaling for Disaggregated LLM Serving with Token Velocity

TL;DR

TokenScale tackles bursty PD-disaggregated LLM serving by introducing Token Velocity, a predictive, stage-unified throughput metric, and Convertible Decoders to absorb bursts with near-zero startup latency. It pairs an offline profiler that estimates per-stage velocities with a control plane that sizes prefillers and decoders based on token arrival rates, while routing bursts to Convertible Decoders for rapid response. Comprehensive experiments across Llama-3.1-8B and Qwen-2.5-32B on A100/H100 show substantial SLO improvements (roughly 80-96% attainment) and 4-14% cost reductions relative to state-of-the-art baselines. The work enables practical, low-latency autoscaling for disaggregated LLM serving and points to future integration with hierarchical KV-cache architectures.

Abstract

The architectural shift to prefill/decode (PD) disaggregation in LLM serving improves resource utilization but struggles with the bursty nature of modern workloads. Existing autoscaling policies, often retrofitted from monolithic systems like those in AIBrix and DistServe, rely on lagging indicators such as GPU utilization or coarse-grained request counts. This results in slow reactions to load spikes, leading to significant Time-to First-Token (TTFT) and Time-Per-Output-Token (TPOT) SLO violations and costly over-provisioning. We introduce TokenScale, an autoscaling framework that resolves this performance mismatch through two innovations. First, we propose Token Velocity, a novel metric that unifies the prefill, network, and decode stages by quantifying their rate of work. As a leading indicator of system backpressure, it enables proactive scaling. Second, Convertible Decoders allow decoder GPUs to dynamically execute prefill tasks during traffic spikes, creating a rapid-response buffer that absorbs bursts and eliminates the initialization latency of new prefillers. Our evaluation on a GPU cluster with production traces shows TokenScale improves SLO attainment from 50-88% to 80-96% and reduces costs by 4-14% over state-of-the-art systems, including DistServe, BlitzScale, and AIBrix. By uniting a predictive metric with a flexible system design, TokenScale significantly boosts the performance and efficiency of disaggregated LLM serving infrastructure.

Figures (15)

• Figure 1: Disaggregated LLM serving system architecture.
• Figure 2: Traffic represented as requests (left) and tokens (right) in a production code trace choukse:dynamollm. Bursts are the spikes above the running average.
• Figure 3: Percentage of burst traffic while varying the overprovisioning ratios in four production traces choukse:dynamollmwang2025burstgpt.
• Figure 4: Utilization of compute (Comp) and memory capacity (Mem) on a single-GPU Prefiller and Decoder instances, and network bandwidth (Net) normalized to their maximum values in the A100 cluster, when serving a burst of requests with Llama-3.1-8B.
• Figure 5: Overview of Token Velocity in prefill, network, and decode stages. $\lambda^{\prime}$ denotes the combined rate of input and predicted output tokens. Token velocity denotes the maximum token processing rate at the prefill ($V_P$), network ($V_N$), and decode ($V_D$) stages. The autoscaler adjusts the number of prefillers ($I^P$) and decoders ($I^D$) based on the ratio between the token arrival rate and the corresponding stage velocity, ensuring that no stage becomes a bottleneck.