Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Exact Hits to Close Enough: Semantic Caching for LLM Embeddings

Dvir David Biton, Roy Friedman

TL;DR

This paper explores offline policies for semantic caching, proving that implementing an optimal offline policy is NP-hard, and proposes several polynomial-time heuristics that combine recency, frequency, and locality.

Abstract

The rapid adoption of large language models (LLMs) has created demand for faster responses and lower costs. Semantic caching, reusing semantically similar requests via their embeddings, addresses this need but breaks classic cache assumptions and raises new challenges. In this paper, we explore offline policies for semantic caching, proving that implementing an optimal offline policy is NP-hard, and propose several polynomial-time heuristics. We also present online semantic aware cache policies that combine recency, frequency, and locality. Evaluations on diverse datasets show that while frequency based policies are strong baselines, our novel variant improves semantic accuracy. Our findings reveal effective strategies for current systems and highlight substantial headroom for future innovation. All code is open source.

From Exact Hits to Close Enough: Semantic Caching for LLM Embeddings

TL;DR

This paper explores offline policies for semantic caching, proving that implementing an optimal offline policy is NP-hard, and proposes several polynomial-time heuristics that combine recency, frequency, and locality.

Abstract

The rapid adoption of large language models (LLMs) has created demand for faster responses and lower costs. Semantic caching, reusing semantically similar requests via their embeddings, addresses this need but breaks classic cache assumptions and raises new challenges. In this paper, we explore offline policies for semantic caching, proving that implementing an optimal offline policy is NP-hard, and propose several polynomial-time heuristics. We also present online semantic aware cache policies that combine recency, frequency, and locality. Evaluations on diverse datasets show that while frequency based policies are strong baselines, our novel variant improves semantic accuracy. Our findings reveal effective strategies for current systems and highlight substantial headroom for future innovation. All code is open source.
Paper Structure (54 sections, 4 theorems, 21 equations, 8 figures, 1 table)

This paper contains 54 sections, 4 theorems, 21 equations, 8 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Summary of Contributions
  4. Preliminaries
  5. Semantic Caching
  6. Embedding
  7. Embeddings Similarity
  8. Theoretical Analysis
  9. OPT
  10. VOPT
  11. Requests.
  12. Request sequence.
  13. Correctness.
  14. VOPT heuristics
  15. Clustered Relaxed Vector Belady
  16. ...and 39 more sections

Key Result

Theorem 3.1

Let $D_{\text{thresh}} > 0$ and let $k$ be the cache capacity. Given a request sequence of vectors, computing VOPT is NP-hard.

Figures (8)

  • Figure 1: Hit rate as a function of cache size - note log scale on the Y-axis
  • Figure 2: Mean hit distance as a function of cache size - note log scale on the Y-axis
  • Figure 3: Point Density Distribution for Various Semantic Distance Threshold Values
  • Figure 4: Throughput as a function of cache size - note log scale on the Y-axis
  • Figure 5: Hit rate at $D_{\text{thresh}} = 0.5$ as a function of cache size - note log scale on the Y-axis
  • ...and 3 more figures

Theorems & Definitions (7)

  • Theorem 3.1: Hardness of Computing VOPT
  • proof
  • Corollary 3.2: Inapproximability of VOPT
  • proof
  • Theorem 3.3: Well-Definedness and Computational Infeasibility of VOPT
  • proof
  • Corollary 3.4: Approximability of the Static Formulation