Randomization Boosts KV Caching, Learning Balances Query Load: A Joint Perspective

TL;DR

The paper tackles the cache-eviction and query-routing tension in KV caching for LLM inference under online, dynamic workloads. It first establishes a formal online model linking per-worker cache state, service time, and queuing to end-to-end latency, then proves that traditional Leaf-LRU eviction can be exponentially suboptimal in worst cases. To address this, it introduces Randomized Leaf Token eviction (RLT), achieving a provably better $O(\log n)$ competitive ratio, and Learning-Based Greedy Routing (LBGR), which online-learns end-to-end latency and greedily routes queries to the best-performing worker. Across four benchmarks and three prefix-sharing settings, the combined approach yields substantial improvements in cache hit rate, latency, TTFT, and throughput, demonstrating practical viability for scalable, dynamic multi-LLM serving. The work offers a principled framework and open-source tooling to guide KV cache design under evolving traffic patterns.

Abstract

KV caching is a fundamental technique for accelerating Large Language Model (LLM) inference by reusing key-value (KV) pairs from previous queries, but its effectiveness under limited memory is highly sensitive to the eviction policy. The default Least Recently Used (LRU) eviction algorithm struggles with dynamic online query arrivals, especially in multi-LLM serving scenarios, where balancing query load across workers and maximizing cache hit rate of each worker are inherently conflicting objectives. We give the first unified mathematical model that captures the core trade-offs between KV cache eviction and query routing. Our analysis reveals the theoretical limitations of existing methods and leads to principled algorithms that integrate provably competitive randomized KV cache eviction with learning-based methods to adaptively route queries with evolving patterns, thus balancing query load and cache hit rate. Our theoretical results are validated by extensive experiments across 4 benchmarks and 3 prefix-sharing settings, demonstrating improvements of up to 6.92$\times$ in cache hit rate, 11.96$\times$ reduction in latency, 14.06$\times$ reduction in time-to-first-token (TTFT), and 77.4% increase in throughput over the state-of-the-art methods. Our code is available at https://github.com/fzwark/KVRouting.

Figures (15)

• Figure 1: Key trade-offs between single LLM cache affinity and global load balancing in the KV cache-aware load balancing problem.
• Figure 2: Results on Llama-3.1-8B-Instruct (Latency, TTFT, and Throughput normalized for comparison). Rows correspond to: GSP (top), ShareGPT (second), UltraChat (third), and Loogle (bottom). For the first four metrics, lower is better; for the last two, higher is better. Our algorithms consistently outperform all baselines across all benchmarks and metrics.
• Figure 3: Results on Llama-3.1-70B-Instruct (top) and Mixtral-8$\times$7B-Instruct-v0.1 (bottom) under the ShareGPT benchmark. Our algorithms consistently outperform all baselines across metrics.
• Figure 4: Results on Llama-3.1-8B-Instruct under the Loogle benchmark with the worst-case round-robin arrival order. Round-robin alternates queries to disrupt KV locality. LBGR+RLT counters this, outperforming all baselines across all metrics.
• Figure 5: Results on Llama-3.1-8B-Instruct under GSP benchmark with varying request rates.

Theorems & Definitions (27)

• Lemma 0
• Lemma 0
• Lemma 0
• Theorem 1: Single-Query
• Theorem 2: Batch
• Theorem 3: Single-Query
• Corollary 3: Batch
• Theorem 4: Single-Query
• Corollary 4: Batch
• Lemma 4