Competitive Non-Clairvoyant KV-Cache Scheduling for LLM Inference
Yiding Feng, Zonghan Yang, Yuhao Zhang
TL;DR
The paper tackles the challenge of scheduling LLM inference when each request’s KV-cache memory grows with decoded tokens, under a hard memory budget and without knowledge of final response lengths. It introduces Geometric-Slicing-Algorithm (GSA), a non-clairvoyant offline batch policy that achieves a constant competitive ratio, by combining Staggered-Pipeline-Schedule (SPS) with geometrically phased execution and a memory-time area analysis. A clairvoyant counterpart, Geometric-Batching-Algorithm (GBA), yields strong approximation guarantees (approximately $10.67$ general and $6.75$ large-memory), serving as a structural benchmark to bound GSA’s performance against the optimum. Numerical experiments on real traces confirm robust practical performance while preserving worst-case guarantees. Overall, the work provides the first constant-competitive non-clairvoyant approach to KV-cache scheduling in LLM inference and offers insights into memory-aware batching via staggered execution and geometric phase structuring.
Abstract
Large Language Model (LLM) inference presents a unique scheduling challenge due to the Key-Value (KV) cache, where a job's memory footprint grows linearly with the number of decoded tokens. This growth couples scheduling decisions with feasibility: a scheduler must minimize latency under a hard memory budget, yet the response lengths of requests are inherently unknown. While recent works have explored this problem either assuming clairvoyance -- exact knowledge of response lengths -- or relying on machine-learned predictions, obtaining robust performance guarantees without any prior knowledge of job sizes remains a theoretically fundamental and practically important open problem. In this work, we propose the Geometric Slicing Algorithm (GSA), the non-clairvoyant policy to achieve the first constant competitive ratio for this problem in the offline batch setting. GSA manages uncertainty through a geometric phase structure that periodically restarts jobs to bound memory exposure, combined with a staggered pipeline mechanism that enables high concurrency by smoothing aggregate memory consumption. We prove that GSA achieves a competitive ratio of at most 61.92 for general instances, improving to 32 in the large-memory regime. Our algorithmic framework also yields a clairvoyant counterpart, the Geometric Batching Algorithm (GBA), which achieves an approximation ratio of 10.67 for general instances and 6.75 in the large-memory regime -- significantly improving upon the best previously known bound of over 9000. Numerical experiments on real request traces demonstrate that our algorithms perform robustly while preserving these worst-case guarantees.