ORBITFLOW: SLO-Aware Long-Context LLM Serving with Fine-Grained KV Cache Reconfiguration

TL;DR

ORBITFLOW tackles memory volatility in long-context LLM serving by introducing fine-grained, per-request KV cache placement guided by a lightweight ILP solver. It continuously adapts KV allocations during decoding using runtime feedback, and employs a token deposit and a pause-resume fallback to preserve system-wide SLOs under heavy load. The approach yields up to 66% and 48% gains in TPOT and TBT SLO attainment, respectively, and up to 3.3x higher throughput with modest runtime overhead, outperforming static offloading baselines across context lengths and distributed settings. This work provides a practical, scalable framework for SLO-aware memory management in disaggregated LLM serving, enabling more robust and responsive long-context inference.

Abstract

Serving long-context LLMs is challenging because request lengths and batch composition vary during token generation, causing the memory footprint to fluctuate significantly at runtime. Offloading KV caches to host memory limits effective memory usage, but existing static and predetermined offloading strategies cannot adapt to the rapidly shifting memory demands of long-context serving. This often leads to excessive CPU-to-GPU KV transfers that translate into latency spikes and frequent SLO violations. To address these challenges, we introduce ORBITFLOW, a fine-grained and adaptive KV cache management system that meets latency SLOs in long-context LLM serving. ORBITFLOW employs a lightweight ILP solver to decide which layers' KV caches to retain on the GPU for each request, within memory capacity constraints. It continuously refines KV placements based on runtime feedback when the active plan becomes suboptimal during token generation. Under heavy load, ORBITFLOW invokes a fallback mechanism to temporarily defer in-flight requests with large memory footprints, preserving overall SLO attainment. Our experiments demonstrate that ORBITFLOW improves SLO attainment for TPOT and TBT by up to 66% and 48%, respectively, while reducing the 95th percentile latency by 38% and achieving up to 3.3x higher throughput compared to existing offloading methods.

Figures (15)

• Figure 1: (a) Prefill stage computes all input tokens in parallel. (b) Decode stage loads KV from the cache and computes only one token. (c) MHA maps distinct KV per query head, while GQA shares KV across head groups to reduce the KV size.
• Figure 2: (a) The KV cache size of various sequence lengths with the LLaMA3-70B model. (b) A layer's computation time and communication time to transfer its KV from CPU to GPU over sequence lengths with the LLaMA3-8B model. (c) With a fixed token-level SLO, tokens generated in early steps become the latency margin.
• Figure 3: An illustrative example showing why a static, uniform offload method is insufficient. The timelines track one decode step early (Step 1, left) and one much later (Step 16, right) for three placements. Each timeline is annotated with the resident KV blocks on GPU, the size of the per-layer prefetch buffer, and the resulting stalls.
• Figure 4: The overall architecture of OrbitFlow.
• Figure 5: (a) Deposited tokens mask all the SLO violations for a single request. (b) Deposited tokens mask the transient latency spikes caused by a short-lived request with a long prompt. S1, E1, S2, and E2 mark the start and end of Request 1 and Request 2.