Tail-Optimized Caching for LLM Inference

TL;DR

Tail latency is a critical bottleneck in LLM inference, motivating prompt caching strategies that prioritize high-tail conversations. The authors introduce Tail-Optimized LRU (T-LRU) and the Tail Excess Latency (TEL) objective, deriving a hindsight-optimal Tail-Optimized Belady policy and an online ET-LRU framework that achieves provable optimality under a stochastic multi-turn model. Empirically, T-LRU achieves substantial tail-latency reductions (e.g., up to ~27-28% for P90; ~24% for P95 TTFT) and large improvements in SLO attainment on WildChat and ShareGPT traces, with only modest median latency impact. The work bridges classical caching theory with modern LLM serving, providing both theoretical guarantees and practical, deployable improvements for tail-latency-aware prompt caching.

Abstract

Prompt caching is critical for reducing latency and cost in LLM inference: OpenAI and Anthropic report up to 50-90% cost savings through prompt reuse. Despite its widespread success, little is known about what constitutes an optimal prompt caching policy, particularly when optimizing tail latency, a metric of central importance to practitioners. The widely used Least Recently Used (LRU) policy can perform arbitrarily poor on this metric, as it is oblivious to the heterogeneity of conversation lengths. To address this gap, we propose Tail-Optimized LRU, a simple two-line modification that reallocates KV cache capacity to prioritize high-latency conversations by evicting cache entries that are unlikely to affect future turns. Though the implementation is simple, we prove its optimality under a natural stochastic model of conversation dynamics, providing the first theoretical justification for LRU in this setting, a result that may be of independent interest to the caching community. Experimentally, on real conversation data WildChat, Tail-Optimized LRU achieves up to 27.5% reduction in P90 tail Time to First Token latency and 23.9% in P95 tail latency compared to LRU, along with up to 38.9% decrease in SLO violations of 200ms. We believe this provides a practical and theoretically grounded option for practitioners seeking to optimize tail latency in real-world LLM deployments.

Figures (7)

• Figure 1: Failure of LRU on Tail Latency. Consider an example with two conversations (A and B) and a total of three turns (A has two turns; B has one). The top panel shows the total job size (conversation history plus user prompt) at each step. For simplicity, we assume the response length is zero and that each user prompt consists of 100 blocks. The bottom panel shows the number of cached blocks updated after each turn. The cache capacity is 100 blocks. We measure tail latency as the maximum processing time among the three requests (i.e., the $66.7\%$-th percentile). Under LRU, all of conversation A's cache blocks are evicted immediately after step 2, resulting in a maximum of 200 uncached blocks at step 3. Therefore, the tail latency under LRU corresponds to the time required to process 200 blocks. In contrast, if at step 2 we partially evict A's cache so that both A and B retain 50 cached tokens each, then regardless of whether the third request comes from A or B, the maximum number of uncached blocks is reduced to 150—an improvement of 25%. This improvement is what our proposed policy, T-LRU, is designed to achieve.
• Figure 2: A three-turn conversation $\{(Q_1,A_1),(Q_2,A_2),(Q_3,A_3)\}$ is shown, $L_i$ denotes conversation history (i.e., $L_1=Q_1+A_1, L_2=L_1+Q_2 + A_2$). Bars indicate the number of uncached blocks each turn incurs; the red dashed line marks the latency threshold $\xi$. Turn 1: projected load for next request is $L_1 + Q_2 <\xi$; no caching is needed. Turn 2: now $L_2 + Q_3 >\xi$, at least $L_2 + Q_3 -\xi$ blocks must be cached (green), but caching more won't improve TEL further.
• Figure 3: Experimental results demonstrate that TTFT increases approximately linearly with the number of uncached tokens. The plot shows TTFT latency as a function of the total prompt length (Token Load) and the size of the cached prefix (Cached Load). We conducted our experiments using Vicuna-7B with vLLM’s prefix caching enabled on a Colab A100 GPU.
• Figure 4: Distributions of turns and tokens of WildChat zhao2024wildchat datasets (sampled 10,000 conversations).
• Figure 5: Latency results for various settings (threshold latency $\xi_{s}$ = 100, 200, 300 ms) from top to bottom panels.

Theorems & Definitions (7)

• Theorem 1: Hindsight-Optimal Policy Structure
• Definition 1: Expected Tailed-Optimized LRU
• Theorem 2: Optimality of Expected-Tail-Optimized LRU
• proof
• proof
• Lemma 1
• proof