QVCache: A Query-Aware Vector Cache

TL;DR

QVCache tackles the core tension in large-scale vector search: achieving high recall under strict latency with limited memory. It introduces a backend-agnostic, query-level similarity cache built from multiple in-memory mini-indexes that store cached vectors and use region-specific, online-learned distance thresholds to decide cache hits. By combining tiered search, adaptive policies, and a Portable online-threshold learning algorithm, QVCache delivers sub-millisecond hit latencies and can reduce end-to-end query latency by up to $\text{60–3000x}$ in practical settings, while maintaining recall close to the backend. The approach is validated across diverse datasets and backends, with experiments showing robust performance under temporal-semantic locality and scalable memory footprints, making it a practical caching layer for scalable vector search in production systems.

Abstract

Vector databases have become a cornerstone of modern information retrieval, powering applications in recommendation, search, and retrieval-augmented generation (RAG) pipelines. However, scaling approximate nearest neighbor (ANN) search to high recall under strict latency SLOs remains fundamentally constrained by memory capacity and I/O bandwidth. Disk-based vector search systems suffer severe latency degradation at high accuracy, while fully in-memory solutions incur prohibitive memory costs at billion-scale. Despite the central role of caching in traditional databases, vector search lacks a general query-level caching layer capable of amortizing repeated query work. We present QVCache, the first backend-agnostic, query-level caching system for ANN search with bounded memory footprint. QVCache exploits semantic query repetition by performing similarity-aware caching rather than exact-match lookup. It dynamically learns region-specific distance thresholds using an online learning algorithm, enabling recall-preserving cache hits while bounding lookup latency and memory usage independently of dataset size. QVCache operates as a drop-in layer for existing vector databases. It maintains a megabyte-scale memory footprint and achieves sub-millisecond cache-hit latency, reducing end-to-end query latency by up to 40-1000x when integrated with existing ANN systems. For workloads exhibiting temporal-semantic locality, QVCache substantially reduces latency while preserving recall comparable to the underlying ANN backend, establishing it as a missing but essential caching layer for scalable vector search.

Figures (15)

• Figure 1: Handling cache hits and misses in QVCache with four mini-indexes, each with a capacity of three vectors. Mini-indexes are ordered by the eviction policy metadata: the leftmost, MI-2, is the hottest (MRU), and the rightmost, MI-3, is the coldest (LRU).
• Figure 2: Nearest Neighbor overlap under perturbation (DEEP deep dataset, $k=10$). The overlap between the neighbor sets of the base and perturbed queries decays sharply, approaching near-zero at a noise ratio of 0.5.
• Figure 3: Evaluation framework proposed in this paper for benchmarking vector caches, used to evaluate QVCache.
• Figure 4: Effect of varying $k$ on the SIFT dataset. Cache capacity is 100,000 for $k = 10$ and is scaled linearly with $k$ (downscaled for $k = 1$ and upscaled for $k = 100$).
• Figure 5: Vector search performance of backend vector database (DiskANN) alone vs. backend augmented with QVCache on the five datasets. $k$ is set to 10.