Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Practical and Asymptotically Optimal Quantization of High-Dimensional Vectors in Euclidean Space for Approximate Nearest Neighbor Search

Jianyang Gao, Yutong Gou, Yuexuan Xu, Yongyi Yang, Cheng Long, Raymond Chi-Wing Wong

TL;DR

This work extends RaBitQ to support moderate compression rates for high-dimensional vector quantization in Approximate Nearest Neighbor (ANN) search. It constructs a codebook from shifting, normalizing, and randomly rotating $B$-bit unsigned integers, preserving the unbiased inner-product estimator and achieving asymptotic optimality in the space–error trade-off. The method is integrated with IVF-based in-memory ANN to enable efficient distance estimation, using a two-stage approach that first exploits the most significant bits for pruning before refining with remaining bits. Comprehensive experiments on real-world datasets show superior recall and efficiency at the same memory footprint compared with state-of-the-art baselines, including strong scalability to ~100M vectors and an empirical validation of the theoretical guarantees.

Abstract

Approximate nearest neighbor (ANN) query in high-dimensional Euclidean space is a key operator in database systems. For this query, quantization is a popular family of methods developed for compressing vectors and reducing memory consumption. Recently, a method called RaBitQ achieves the state-of-the-art performance among these methods. It produces better empirical performance in both accuracy and efficiency when using the same compression rate and provides rigorous theoretical guarantees. However, the method is only designed for compressing vectors at high compression rates (32x) and lacks support for achieving higher accuracy by using more space. In this paper, we introduce a new quantization method to address this limitation by extending RaBitQ. The new method inherits the theoretical guarantees of RaBitQ and achieves the asymptotic optimality in terms of the trade-off between space and error bounds as to be proven in this study. Additionally, we present efficient implementations of the method, enabling its application to ANN queries to reduce both space and time consumption. Extensive experiments on real-world datasets confirm that our method consistently outperforms the state-of-the-art baselines in both accuracy and efficiency when using the same amount of memory.

Practical and Asymptotically Optimal Quantization of High-Dimensional Vectors in Euclidean Space for Approximate Nearest Neighbor Search

TL;DR

This work extends RaBitQ to support moderate compression rates for high-dimensional vector quantization in Approximate Nearest Neighbor (ANN) search. It constructs a codebook from shifting, normalizing, and randomly rotating -bit unsigned integers, preserving the unbiased inner-product estimator and achieving asymptotic optimality in the space–error trade-off. The method is integrated with IVF-based in-memory ANN to enable efficient distance estimation, using a two-stage approach that first exploits the most significant bits for pruning before refining with remaining bits. Comprehensive experiments on real-world datasets show superior recall and efficiency at the same memory footprint compared with state-of-the-art baselines, including strong scalability to ~100M vectors and an empirical validation of the theoretical guarantees.

Abstract

Approximate nearest neighbor (ANN) query in high-dimensional Euclidean space is a key operator in database systems. For this query, quantization is a popular family of methods developed for compressing vectors and reducing memory consumption. Recently, a method called RaBitQ achieves the state-of-the-art performance among these methods. It produces better empirical performance in both accuracy and efficiency when using the same compression rate and provides rigorous theoretical guarantees. However, the method is only designed for compressing vectors at high compression rates (32x) and lacks support for achieving higher accuracy by using more space. In this paper, we introduce a new quantization method to address this limitation by extending RaBitQ. The new method inherits the theoretical guarantees of RaBitQ and achieves the asymptotic optimality in terms of the trade-off between space and error bounds as to be proven in this study. Additionally, we present efficient implementations of the method, enabling its application to ANN queries to reduce both space and time consumption. Extensive experiments on real-world datasets confirm that our method consistently outperforms the state-of-the-art baselines in both accuracy and efficiency when using the same amount of memory.
Paper Structure (28 sections, 6 theorems, 12 equations, 7 figures, 3 tables, 1 algorithm)

This paper contains 28 sections, 6 theorems, 12 equations, 7 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. ANN Query
  4. The Quantization Method RaBitQ
  5. Extending RaBitQ
  6. Motivations and Overview
  7. Quantizing Data Vectors
  8. Constructing a Codebook
  9. Computing the Quantization Codes of Data Vectors
  10. Computing the Estimator
  11. Summary and Theoretical Analysis
  12. Applying The Extended RaBitQ to in-memory ANN
  13. Using the Extended RaBitQ with IVF
  14. Conducting Distance Comparison with the Extended RaBitQ
  15. experiments
  16. ...and 13 more sections

Key Result

lemma 1

$\frac{\left< \mathbf{\bar{o}}_0, \mathbf{q} \right>}{\left< \mathbf{\bar{o}}_0, \mathbf{o} \right>}$ is an unbiased estimator of $\left< \mathbf{o,q}\right>$. With the probability of at least $1 - \exp(-c_0\epsilon_0^2)$, its error bound is presented as where $c_0$ is a constant and $\epsilon_0$ is a parameter which controls the failure probability of the bound.

Figures (7)

  • Figure 1: This figure illustrates the quantization codebook of our method when $B=2$ in the 2-dimensional space. The empty blue points in the left panel represent the set $\mathcal{G}$, i.e., a set of vectors on a uniform grid. The solid red points in the right panel represent the normalized vectors in $\mathcal{G}$. Applying a random rotation on the red points yields the codebook $\mathcal{G}_r$.
  • Figure 2: Decomposition of the Quantization Code $\mathbf{\bar{y}}_u$.
  • Figure 3: Space-Accuracy Trade-Off for Distance Estimation (Log-Scale).
  • Figure 4: Time-Accuracy Trade-Off for the ANN Query (Log-Scale), $K=100$. All the methods are combined with the IVF index.
  • Figure 5: Verification Study for Scalability.
  • ...and 2 more figures

Theorems & Definitions (7)

  • lemma 1: Restating Theorem 3.2 in rabitq
  • lemma 2
  • theorem 1
  • Remark : Empirical Formula
  • lemma 3
  • lemma 4
  • lemma 5