CascadeInfer: Low-Latency and Load-Balanced LLM Serving via Length-Aware Scheduling

TL;DR

CascadeInfer addresses the bottleneck created by length heterogeneity in MILS for long-context LLMs by forming a length-aware multi-stage pipeline across identical replicas. It uses a DP-based pipeline planning approach, runtime adaptive range refinement, and a decentralized bid-ask load-balancing protocol to route and migrate requests as sequence length grows, without altering intra-instance schedulers. The method achieves up to $67\%$ end-to-end latency reduction, $69\%$ tail-latency reduction, and up to $2.89\\times$ throughput improvements on a 16-GPU cluster, demonstrating substantial gains in both latency and throughput while maintaining compatibility with existing engines. These results suggest that elevating length heterogeneity to a cluster-level scheduling problem can unlock near-hardware-limit efficiency for production LLM serving.

Abstract

Efficiently harnessing GPU compute is critical to improving user experience and reducing operational costs in large language model (LLM) services. However, current inference engine schedulers overlook the attention backend's sensitivity to request-length heterogeneity within a batch. As state-of-the-art models now support context windows exceeding 128K tokens, this once-tolerable inefficiency has escalated into a primary system bottleneck, causing severe performance degradation through GPU underutilization and increased latency. We present CascadeInfer, a runtime system that dynamically reschedules requests across multiple instances serving the same LLM to mitigate per-instance length heterogeneity. CascadeInfer partitions these instances into length-specialized groups, each handling requests within a designated length range, naturally forming a pipeline as requests flow through them. CascadeInfer devises a dynamic programming algorithm to efficiently find the stage partition with the best QoE, employs runtime range refinement together with decentralized load (re)balance both across and within groups, achieving a balanced and efficient multi-instance service. Our evaluation shows that, under the same configuration, CascadeInfer reduces end-to-end latency by up to 67% and tail latency by up to 69%, while improving overall system throughput by up to 2.89 times compared to the state-of-the-art multi-instance scheduling systems.

Figures (16)

• Figure 1: Request-length distribution in batches under various scheduling policies and request rates. Batches were sampled at 20%, 40%, 60%, and 80% of the inference process. The inputs come from an LLM dialogue dataset shareGPT, and requests longer than 128K are discarded.
• Figure 2: Effect of sequence length heterogeneity on decoding forward pass performance. Measured on a single H100 GPU using vLLM and SGLang with FlashAttention, FlashInfer, and Triton (model: Llama-3.2-3B, batch size: 512).
• Figure 3: Architecture and workflow of CascadeInfer. Engine instances are grouped by length into stages forming a logical pipeline; sequences may exit early without traversing all stages.
• Figure 4: Pipeline planning based on the request length distribution.
• Figure 5: Illustration of intra‑stage load balancing using dynamic decentralized bid-ask scheduling.