Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient Data-aware Distance Comparison Operations for High-Dimensional Approximate Nearest Neighbor Search

Liwei Deng, Penghao Chen, Ximu Zeng, Tianfu Wang, Yan Zhao, Kai Zheng

TL;DR

This work targets the bottleneck in high-dimensional AKNN: distance comparison operations. It introduces DADE, a data-aware distance estimation framework that rotates data with a PCA-based orthogonal transform and estimates distances in a dynamically chosen subspace, controlled by a hypothesis-test-driven expansion. DADE provides an unbiased distance estimator with reduced variance and integrates as a plug-in into AKNN systems like HNSW and IVF, yielding substantial QPS gains while preserving recall (e.g., >40% improvements on the DEEP dataset). The key theoretical contributions include an unbiasedness guarantee with respect to data distribution and a variance-optimized estimation, alongside a practical adaptive mechanism to determine the required projection dimensionality. These results offer a practical, principled path to accelerate AKNN search in large-scale, high-dimensional settings.

Abstract

High-dimensional approximate $K$ nearest neighbor search (AKNN) is a fundamental task for various applications, including information retrieval. Most existing algorithms for AKNN can be decomposed into two main components, i.e., candidate generation and distance comparison operations (DCOs). While different methods have unique ways of generating candidates, they all share the same DCO process. In this study, we focus on accelerating the process of DCOs that dominates the time cost in most existing AKNN algorithms. To achieve this, we propose an Data-Aware Distance Estimation approach, called DADE, which approximates the exact distance in a lower-dimensional space. We theoretically prove that the distance estimation in DADE is unbiased in terms of data distribution. Furthermore, we propose an optimized estimation based on the unbiased distance estimation formulation. In addition, we propose a hypothesis testing approach to adaptively determine the number of dimensions needed to estimate the exact distance with sufficient confidence. We integrate DADE into widely-used AKNN search algorithms, e.g., IVF and HNSW, and conduct extensive experiments to demonstrate the superiority.

Efficient Data-aware Distance Comparison Operations for High-Dimensional Approximate Nearest Neighbor Search

TL;DR

This work targets the bottleneck in high-dimensional AKNN: distance comparison operations. It introduces DADE, a data-aware distance estimation framework that rotates data with a PCA-based orthogonal transform and estimates distances in a dynamically chosen subspace, controlled by a hypothesis-test-driven expansion. DADE provides an unbiased distance estimator with reduced variance and integrates as a plug-in into AKNN systems like HNSW and IVF, yielding substantial QPS gains while preserving recall (e.g., >40% improvements on the DEEP dataset). The key theoretical contributions include an unbiasedness guarantee with respect to data distribution and a variance-optimized estimation, alongside a practical adaptive mechanism to determine the required projection dimensionality. These results offer a practical, principled path to accelerate AKNN search in large-scale, high-dimensional settings.

Abstract

High-dimensional approximate nearest neighbor search (AKNN) is a fundamental task for various applications, including information retrieval. Most existing algorithms for AKNN can be decomposed into two main components, i.e., candidate generation and distance comparison operations (DCOs). While different methods have unique ways of generating candidates, they all share the same DCO process. In this study, we focus on accelerating the process of DCOs that dominates the time cost in most existing AKNN algorithms. To achieve this, we propose an Data-Aware Distance Estimation approach, called DADE, which approximates the exact distance in a lower-dimensional space. We theoretically prove that the distance estimation in DADE is unbiased in terms of data distribution. Furthermore, we propose an optimized estimation based on the unbiased distance estimation formulation. In addition, we propose a hypothesis testing approach to adaptively determine the number of dimensions needed to estimate the exact distance with sufficient confidence. We integrate DADE into widely-used AKNN search algorithms, e.g., IVF and HNSW, and conduct extensive experiments to demonstrate the superiority.

Paper Structure

This paper contains 15 sections, 5 theorems, 9 equations, 5 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Methodology
  4. Unbiased Estimation
  5. Optimized Estimation
  6. Dimension Expansion with Hypothesis Testing
  7. DADE Summarization
  8. Experiment
  9. Experimental Settings
  10. Experimental Results
  11. Overall Performance.
  12. Feasibility of Distance Estimation Methods for DCOs.
  13. Sensitivity Study.
  14. Related Works
  15. Conclusion

Key Result

Lemma 1

Assume $X_1,X_2 \in \mathbb{R}^D$ are independent and identically distributed random vectors, and $Y_i=X_i-\mathbb{E}[X_i]\ (for\ i=1,2)$. For $W_D \in \mathbb{R}^{D\times D}, {W_D}^T W_D=\mathbf{I}$, the following equation holds:

Figures (5)

  • Figure 1: Running Example on DEEP.
  • Figure 2: Time-Recall Tradeoff.
  • Figure 3: Feasibility for DCOs in terms of Recall and QPS.
  • Figure 4: Parameter Study on $p$ of AKNN** Algorithms with Different $K$.
  • Figure 5: Parameter Study on $\Delta_d$ of AKNN** Algorithms with Different $K$.

Theorems & Definitions (6)

  • Lemma 1
  • Lemma 2
  • Definition 1: Distance Comparison Operation gao2023high
  • Lemma 3
  • Lemma 4
  • Lemma 5