vCache: Verified Semantic Prompt Caching

TL;DR

This paper proposes vCache, the first verified semantic cache with user-defined error rate guarantees for predictable performance, which employs an online learning algorithm to estimate an optimal threshold for each cached prompt, enabling reliable cache responses without additional training.

Abstract

Semantic caches return cached responses for semantically similar prompts to reduce LLM inference latency and cost. They embed cached prompts and store them alongside their response in a vector database. Embedding similarity metrics assign a numerical score to quantify the similarity between a request and its nearest neighbor prompt from the cache. Existing systems use the same static similarity threshold across all requests to determine whether two prompts can share similar responses. However, we observe that static thresholds do not give formal correctness guarantees, result in unexpected error rates, and lead to suboptimal cache hit rates. This paper proposes vCache, the first verified semantic cache with user-defined error rate guarantees for predictable performance. It employs an online learning algorithm to estimate an optimal threshold for each cached prompt, enabling reliable cache responses without additional training. Our experiments show that vCache consistently meets the specified error bounds while outperforming state-of-the-art static-threshold and fine-tuned embedding baselines with up to 12.5$\times$ higher cache hit and 26$\times$ lower error rates. We release the vCache implementation and four benchmarks to support future research.

Figures (15)

• Figure 1: The static threshold in existing semantic caches enforces naive decision boundaries, resulting in either low cache hit or high error rates. vCache's embedding-specific and dynamic thresholds learn decision boundaries to guarantee a user-defined maximum error rate. Gradient shading reflects decreasing confidence in correctness as similarity to the cached embedding decreases.
• Figure 2: Exploration probability for $emb_{42}$ and $emb_{128}$. Top: Observations are perfectly separable. Bottom: Observations are overlapping. vCache selects the optimal $\hat{t}$ and adjusts the exploration probability based on the user-defined $\delta=0.01$.
• Figure 3: Results from 45k samples in the SemCacheClassification benchmark. Motivates the need for dynamic, embedding-specific thresholds. Top: Similarity values of correct and incorrect explorations exhibit highly overlapping distributions. Bottom: Optimal per-embedding thresholds vary substantially, indicating that no single threshold can suffice across embeddings (see Appendix \ref{['app:threshold_dilemma']}).
• Figure 4: vCache meets the user-defined maximum error rate bound $\delta$ with steadily increasing cache hit rates (vCache is learning). GPTCache shows increasing error and hit rates, illustrating the unreliability of static thresholds. Static baselines use fixed thresholds. See Figure \ref{['fig:dynamic_emb_specific_thresholds']} for a threshold vs. $\delta$ Pareto comparison.
• Figure 5: Pareto comparison across a range of thresholds and $\delta$ values. Each point represents a full run on all samples of a dataset.

Theorems & Definitions (6)

• Definition 4.1: user-guarantee
• Theorem 4.1
• Lemma C.1: Upper-Bounding $\tau$
• proof
• Lemma C.2: Lower‐Bounding Cache‐Correctness Probability
• proof