From Tokens to Layers: Redefining Stall-Free Scheduling for LLM Serving with Layered Prefill

TL;DR

This paper proposes layered prefill, a scheduling paradigm that partitions a decoder stack into contiguous layer groups and performs prefill for one group per iteration, eliminating repeated MoE weight reloads caused by token-level chunking. By shifting the scheduling axis from tokens to layers, it preserves stall-free decoding while reducing off-chip memory traffic and energy, achieving up to $TTFT$ improvements up to 70% and end-to-end latency reductions of about 41% in long-context workloads. Across two MoE models and two datasets, expert-load traffic drops by 12–39% and energy per output token by 8–22%, with higher sustainable throughput under strict SLOs. This approach demonstrates a practical, architecture-aware path to more energy-efficient, scalable LLM serving in co-located data-center environments.

Abstract

Large Language Model (LLM) inference in production must meet stringent service-level objectives for both time-to-first-token (TTFT) and time-between-token (TBT) while maximizing throughput under fixed compute, memory, and interconnect budgets. Modern serving systems adopt stall-free scheduling techniques such as chunked prefill, which splits long prompt processing along the token dimension and interleaves prefill with ongoing decode iterations. While effective at stabilizing TBT, chunked prefill incurs substantial overhead in Mixture-of-Experts (MoE) models: redundant expert weight loads increase memory traffic by up to 39% and inflate energy consumption. We propose layered prefill, a new scheduling paradigm that treats transformer layer groups as the primary scheduling unit. By vertically partitioning the model into contiguous layer groups and interleaving prefill and decode across the groups, layered prefill sustains stall-free decoding while eliminating chunk-induced MoE weight reloads. It reduces off-chip bandwidth demand, lowering TTFT by up to 70%, End-to-End latency by 41% and per-token energy by up to 22%. Evaluations show that layered prefill consistently improves the TTFT--TBT Pareto frontier over chunked prefill, reducing expert-load traffic and energy cost while maintaining stall-free decoding. Overall, shifting the scheduling axis from tokens to layers unlocks a new operating regime for high-efficiency, energy-aware LLM serving in co-located environments.

Figures (5)

• Figure 1: (Upper right) Per iteration, chunked prefill splits an input prompt into multiple chunks, and at each iteration one chunk is processed in order from the beginning with the decode. (Lower right) For layered prefill, exactly one layer group performs both prefill and decode, while the others perform decode only. Prefill advances by one group per iteration, maintaining stall-free decoding.
• Figure 2: (Left) MoE weight loading vs. chunk size. The hatched region indicates the MoE weights loaded by a single chunk. (Right) runtime of each kernel vs. chunk size. We fix the input length fixed at 8,192 tokens.
• Figure 3: SLO attainment under different request rates. The red horizontal line marks the effective SLO attainment threshold (90%).
• Figure 4: Breakdown of SLO attainment by component across request rates.
• Figure 5: Token generation over time on arXiv with Qwen.