Locally-Adaptive Quantization for Streaming Vector Search

TL;DR

This paper investigates Locally-Adaptive Vector Quantization (LVQ) for streaming similarity search and introduces Turbo LVQ and Multi-Means LVQ (M-LVQ) to improve speed and accuracy in dynamic databases. It analyzes how quantization errors map to search performance, showing LVQ remains robust under data distribution shifts and that both Turbo LVQ and M-LVQ yield substantial, dataset-dependent speedups. The proposed SVS-LVQ system achieves state-of-the-art throughput in IID and distribution-shift scenarios, outperforming FreshVamana and HNSWlib by up to 9.4x and 8.8x, respectively, while maintaining stable recall and efficient index updates. The authors release open-source code and a new dataset for streaming evaluation, underscoring practical impact for real-world retrieval systems and RAG-style pipelines that require fast, up-to-date vector search.

Abstract

Retrieving the most similar vector embeddings to a given query among a massive collection of vectors has long been a key component of countless real-world applications. The recently introduced Retrieval-Augmented Generation is one of the most prominent examples. For many of these applications, the database evolves over time by inserting new data and removing outdated data. In these cases, the retrieval problem is known as streaming similarity search. While Locally-Adaptive Vector Quantization (LVQ), a highly efficient vector compression method, yields state-of-the-art search performance for non-evolving databases, its usefulness in the streaming setting has not been yet established. In this work, we study LVQ in streaming similarity search. In support of our evaluation, we introduce two improvements of LVQ: Turbo LVQ and multi-means LVQ that boost its search performance by up to 28% and 27%, respectively. Our studies show that LVQ and its new variants enable blazing fast vector search, outperforming its closest competitor by up to 9.4x for identically distributed data and by up to 8.8x under the challenging scenario of data distribution shifts (i.e., where the statistical distribution of the data changes over time). We release our contributions as part of Scalable Vector Search, an open-source library for high-performance similarity search.

Figures (26)

• Figure 1: The newly introduced open-images-512-13M dataset (\ref{['ssec:datasets_and_procols']}) contains vectors organized in semantical classes that have different distributions, as evidenced by the 2D Principal Component Analysis projections of vectors sampled from four classes (left). For example, vectors from fountain are close to those from building as these images often display buildings. On the right, we show a few examples kuznetsova_open_2020 from each of the four classes: box (top-left), building (top-right), food (bottom-left), and fountain (bottom-right).
• Figure 2: LVQ uses the original dimension ordering to store each vector. Turbo LVQ permutes the dimensions of the database vectors to make their decoding more efficient. Here, we show an example where the permutation is applied in groups of 128 dimensions, using 4 bits per dimension bundled in sixteen 32-bit words.
• Figure 3: Heatmap showing the multiplicative increment in search window size (ratio w.r.t using float32 encoded vectors) required to reach 0.9 10 recall@10 for the open-images-512-1M dataset when using LVQ-$B_1\mathsf{x}B_2$ compressed vectors. Different combinations of the number of bits for the first ($B_1$) and second ($B_2$) LVQ-levels represent different noise level scenarios. For large enough $B_1$ and $B_2$ (lower-right corner), there is no search window size increase with respect to using float32 vectors, whereas for small enough $B_1$ and $B_2$ the target recall cannot be reached regardless of the search window size increase (the increment is not reported for these cells). The required window size strongly depends on the reconstruction error in the first level (represented by $B_1$) because there is only so much the re-ranking can fix when having a bad set of neighbor candidates. On the contrary, for a given $B_1$, the quantization error at the second LVQ level only impacts the search window size for very small $B_2$ ($B_2\leq 3$).
• Figure 4: Search window size ratio (with respect to using float32-encoded vectors) required to reach 0.9 10 recall@10 on open-images-512-1M (\ref{['ssec:datasets_and_procols']}) as a function of the average first-level quantization error. We use $B_2=8$, which corresponds to the column $B_2=8$ in \ref{['fig:quant_error_heatmap']}, and replace $B_1$ by its corresponding average quantization error using \ref{['eq:mean_quant_error_first']} for LVQ and M-LVQ. The vertical lines mark the reconstruction error achieved by LVQ4x8 and M-LVQ4x8 with different number of means, showing that the required search window size can be considerably reduced by M-LVQ if using 10 to 100 means.
• Figure 5: Search window size ratio (with respect to using float32-encoded vectors) required to reach 0.9 10 recall@10 as a function of the number of bits used for the first quantization level (with $B_2=8$) for various datasets. The behavior varies greatly with the dataset when working on the high quantization error regime (i.e. when using a small number of bits $B_1<5$): open-images-512-1M can achieve a large window size reduction by reducing the quantization error, whereas the effect for rqa-768-1M-ID would be very small. For all datasets, the search window size will not be reduced if going beyond 6 bits.

Theorems & Definitions (1)

• Definition 1