Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mitigating KV Cache Competition to Enhance User Experience in LLM Inference

Haiying Shen, Tanmoy Sen, Masahiro Tanaka

TL;DR

CacheOPT tackles the KV-cache bottleneck in LLM inference by combining confidence-based padding, SLO-aware batching, proactive KVC allocation, and adaptive preemption decisions. It uses Hoeffding bounds to bound padding, reuses allocated KVC via embedding, and selects preemption targets and strategies to minimize tail latency while preserving SLOs. Empirical results show substantial improvements in tail TTFT and TBT, higher SLO attainment, and better throughput across multiple models and traces. The work provides a practical, modular approach to KV-cache management with clear guidance on parameter settings and trade-offs for real-world deployment.

Abstract

In Large Language Model (LLM) serving, the KV-cache (KVC) bottleneck causes high tail Time-to-First-Token (TTFT) and Time-Between-Tokens (TBT), impairing user experience, particularly in time-sensitive applications. However, satisfying both TTFT and TBT service-level objectives (SLOs) is challenging. To address this, we propose a system, named CacheOPT for mitigating KV Cache competition, based on key insights from our measurements, incorporating novel components. First, it estimates a request's output length, bounding the deviation with a high specified probability, adjusted based on the request arrival rate. Second, it allocates the estimated KVC demand to a request, and reuses other requests' allocated KVC to avoid preemptions while reducing waiting time. Third, it proactively allocates KVC before instead of at the time a request exhausts its allocation and reserves KVC globally to prevent preemptions. Fourth, it chooses a request that has long TBT SLO, long job remaining time and short preemption time to preempt. Fifth, it selects the shortest-latency strategy between swapping and recomputation for preemptions. Experiments show that CacheOPT achieves up to 3.29$\times$ and 2.83$\times$ lower tail TBT and tail TTFT, 47\% and 53\% higher TTFT and TBT SLO attainments, and supports up to 1.58$\times$ higher request arrival rate than the state-of-the-art methods.

Mitigating KV Cache Competition to Enhance User Experience in LLM Inference

TL;DR

CacheOPT tackles the KV-cache bottleneck in LLM inference by combining confidence-based padding, SLO-aware batching, proactive KVC allocation, and adaptive preemption decisions. It uses Hoeffding bounds to bound padding, reuses allocated KVC via embedding, and selects preemption targets and strategies to minimize tail latency while preserving SLOs. Empirical results show substantial improvements in tail TTFT and TBT, higher SLO attainment, and better throughput across multiple models and traces. The work provides a practical, modular approach to KV-cache management with clear guidance on parameter settings and trade-offs for real-world deployment.

Abstract

In Large Language Model (LLM) serving, the KV-cache (KVC) bottleneck causes high tail Time-to-First-Token (TTFT) and Time-Between-Tokens (TBT), impairing user experience, particularly in time-sensitive applications. However, satisfying both TTFT and TBT service-level objectives (SLOs) is challenging. To address this, we propose a system, named CacheOPT for mitigating KV Cache competition, based on key insights from our measurements, incorporating novel components. First, it estimates a request's output length, bounding the deviation with a high specified probability, adjusted based on the request arrival rate. Second, it allocates the estimated KVC demand to a request, and reuses other requests' allocated KVC to avoid preemptions while reducing waiting time. Third, it proactively allocates KVC before instead of at the time a request exhausts its allocation and reserves KVC globally to prevent preemptions. Fourth, it chooses a request that has long TBT SLO, long job remaining time and short preemption time to preempt. Fifth, it selects the shortest-latency strategy between swapping and recomputation for preemptions. Experiments show that CacheOPT achieves up to 3.29 and 2.83 lower tail TBT and tail TTFT, 47\% and 53\% higher TTFT and TBT SLO attainments, and supports up to 1.58 higher request arrival rate than the state-of-the-art methods.

Paper Structure

This paper contains 25 sections, 6 equations, 17 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Experimental Analysis
  3. Experiment Settings
  4. Impact of KVC Allocation Methods on TTFT and TBT
  5. Padding Size Determination and Impact
  6. Preemption Policy
  7. Preemption Strategy: Swapping or Recomputation?
  8. System Design
  9. Solution Overview
  10. Confidence-based Padding
  11. SLO-aware Batching and KVC Allocation
  12. Embedding Method
  13. Request Selection and KVC Allocation
  14. Proactive KVC Allocation and Global Reservation
  15. Preemption Policy
  16. ...and 10 more sections

Figures (17)

  • Figure 1: Measurements for different traces for OPT-13B.
  • Figure 2: Measurements for different traces for OPT-175B.
  • Figure 3: Average requests added per iteration.
  • Figure 4: Illustration of resource bottleneck.
  • Figure 5: CDF of requests vs. over/under provisioned tokens.
  • ...and 12 more figures