KVShare: An LLM Service System with Efficient and Effective Multi-Tenant KV Cache Reuse

TL;DR

KVShare tackles the high TTFT and compute cost of ultra-long-context LLM serving by enabling cross-request KV cache reuse in multi-tenant settings. It introduces adaptive-length token chunking, a Dual Stage High Deviation (DHD) selector, and a cache-aware scheduler to balance accuracy and efficiency during prefill and decoding. The approach mitigates attention deviations and decoding drift while maximizing KV cache reuse, supported by experiments across Qwen2.5-7B, Llama3.1-8B, and Yi1.5-9B that show substantial TTFT reductions, improved throughput, and competitive accuracy gains versus state-of-the-art methods. These results demonstrate KVShare’s potential to enable efficient, low-latency, multi-tenant LLM services at scale.

Abstract

Recent advances in long-text understanding have pushed the context length of large language models (LLMs) up to one million tokens. It boosts LLMs's accuracy and reasoning capacity but causes exorbitant computational costs and unsatisfactory Time to First Token (TTFT). KV cache reuse, which reuses the exact same KV cache of prefixes and templates or shares similar ones but with extra selective recomputation, offers a promising way to tackle this issue. However, prior studies overlook the cross-request KV reuse and the attention deviations introduced by new tokens during the decoding stage. In this paper, we present a KV cache management module that shares the KV cache across requests under multi-tenant scenarios without sacrificing model accuracy. Our system, KVShare, enables accurate and efficient LLM serving by 1) a Dual-Stage High Deviation algorithm (DHD) that conditionally selects a small portion of KV cache to be recomputed during both prefill and decode phases, and 2) a cache-aware scheduler that prioritizes requests based on their KV cache hit rates and orchestrates continuous batching to achieve enhanced system efficiency and faster TTFT. Multi-task experiments conducted on models such as Qwen2.5-7B,Llama3.1-8B and Yi1.5-9B demonstrate that KVShare reduces TTFT by up to 9.39x and increases 1.2x of the throughput compared to the full KV recompute. Moreover, KVShare achieves 20.38% boost in terms of accuracy compared to SOTA methods.

Figures (11)

• Figure 1: The relation between chunk size and accuracy & hit rate. The results are profiled with Llama3.1-8B in SAMSum dataset.
• Figure 2: KVShare system for low-latency context retrieval and adaptive token re-computation.
• Figure 3: Comparison of hit rates for KVShare, CacheBlend, and EPIC methods across four chat datasets: ShareGPT ShareGPT-Chinese-English-90k, InstructionV2 instructionwild, LMSysinstructionwild, and WildChat zhao2024wildchat.
• Figure 4: In Figure\ref{['fig:recompute_strategy_comparison']}, we use the Llama3.1-8B model on the GSM8K dataset to test the impact of different token selection strategies on the error of $\Delta H$. The results show that the ideal index method and our proposed method have similar effects, both of which can minimize the error of $\Delta H$ to the greatest extent. In Figure\ref{['fig:attention_shift']}, it is evident that there is a significant shift in the attention during the decode phase compared to the cross - attention in the prefill phase. This indicates that some tokens with relatively low attention scores in the prefill phase receive higher attention during the decode phase. We visualized this using the attention data from 12 heads in Layer 25 of Llama 3.1.
• Figure 5: Cache-Aware scheduling optimizes Transformer batch processing performance.Figure\ref{['fig:exec_batch_hit_a']}: Comparison of FCFS and Cache-Aware scheduling: FCFS causes significant hit rate disparities in the same batch (e.g., R1/R3 vs. R2/R4), while Cache-Aware accelerates TTFT by prioritizing high-hit-rate requests and merging similar ones. Figure\ref{['fig:schedule_demo']}: The gap between actual and ideal performance curves shows Cache-Aware’s TTFT optimization potential (tested on Llama3.1-8b with GSM8K).