Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fast Tuning the Index Construction Parameters of Proximity Graphs in Vector Databases

Wenyang Zhou, Jiadong Xie, Yingfan Liu, Zhihao Yin, Jeffrey Xu Yu, Hui Li, Zhangqian Mu, Xiaotian Qiao, Jiangtao Cui

TL;DR

FastPGT tackles the costly problem of tuning proximity-graph construction parameters for $k$-ANNS by enabling batch parameter recommendations and simultaneous construction of multiple PGs to exploit shared computations. It eliminates the RNG parameter $R$ from tuning, introduces a batch MOBO approach via $m$EHVI, and deploys efficient shared Search and Prune strategies (via $V_{\delta}$ caching and $mKANNS$/ $mPrune$) to dramatically reduce distance computations. The framework is model-agnostic and PG-agnostic, demonstrated to achieve up to $2.37\times$ speedups over the previous SOTA (VDTuner) while maintaining or improving Recall$@k$ and QPS across HNSW, NSG, and Vamana on real-world datasets. These contributions significantly accelerate practical tuning of PG indexes for vector databases and retrieval-augmented systems, enabling faster deployment of high-performance $k$-NN search pipelines.

Abstract

k-approximate nearest neighbor search (k-ANNS) in high-dimensional vector spaces is a fundamental problem across many fields. With the advent of vector databases and retrieval-augmented generation, k-ANNS has garnered increasing attention. Among existing methods, proximity graphs (PG) based approaches are the state-of-the-art (SOTA) methods. However, the construction parameters of PGs significantly impact their search performance. Before constructing a PG for a given dataset, it is essential to tune these parameters, which first recommends a set of promising parameters and then estimates the quality of each parameter by building the corresponding PG and then testing its k-ANNS performance. Given that the construction complexity of PGs is superlinear, building and evaluating graph indexes accounts for the primary cost of parameter tuning. Unfortunately, there is currently no method considered and optimized this process.In this paper, we introduce FastPGT, an efficient framework for tuning the PG construction parameters. FastPGT accelerates parameter estimation by building multiple PGs simultaneously, thereby reducing repeated computations. Moreover, we modify the SOTA tuning model to recommend multiple parameters at once, which can be efficiently estimated using our method of building multiple PGs simultaneously. Through extensive experiments on real-world datasets, we demonstrate that FastPGT achieves up to 2.37x speedup over the SOTA method VDTuner, without compromising tuning quality.

Fast Tuning the Index Construction Parameters of Proximity Graphs in Vector Databases

TL;DR

FastPGT tackles the costly problem of tuning proximity-graph construction parameters for -ANNS by enabling batch parameter recommendations and simultaneous construction of multiple PGs to exploit shared computations. It eliminates the RNG parameter from tuning, introduces a batch MOBO approach via EHVI, and deploys efficient shared Search and Prune strategies (via caching and / ) to dramatically reduce distance computations. The framework is model-agnostic and PG-agnostic, demonstrated to achieve up to speedups over the previous SOTA (VDTuner) while maintaining or improving Recall and QPS across HNSW, NSG, and Vamana on real-world datasets. These contributions significantly accelerate practical tuning of PG indexes for vector databases and retrieval-augmented systems, enabling faster deployment of high-performance -NN search pipelines.

Abstract

k-approximate nearest neighbor search (k-ANNS) in high-dimensional vector spaces is a fundamental problem across many fields. With the advent of vector databases and retrieval-augmented generation, k-ANNS has garnered increasing attention. Among existing methods, proximity graphs (PG) based approaches are the state-of-the-art (SOTA) methods. However, the construction parameters of PGs significantly impact their search performance. Before constructing a PG for a given dataset, it is essential to tune these parameters, which first recommends a set of promising parameters and then estimates the quality of each parameter by building the corresponding PG and then testing its k-ANNS performance. Given that the construction complexity of PGs is superlinear, building and evaluating graph indexes accounts for the primary cost of parameter tuning. Unfortunately, there is currently no method considered and optimized this process.In this paper, we introduce FastPGT, an efficient framework for tuning the PG construction parameters. FastPGT accelerates parameter estimation by building multiple PGs simultaneously, thereby reducing repeated computations. Moreover, we modify the SOTA tuning model to recommend multiple parameters at once, which can be efficiently estimated using our method of building multiple PGs simultaneously. Through extensive experiments on real-world datasets, we demonstrate that FastPGT achieves up to 2.37x speedup over the SOTA method VDTuner, without compromising tuning quality.
Paper Structure (18 sections, 2 theorems, 3 equations, 9 figures, 6 tables, 6 algorithms)

This paper contains 18 sections, 2 theorems, 3 equations, 9 figures, 6 tables, 6 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. PG Construction Methods
  4. Problem Statement
  5. Existing Methods and Their Issues
  6. Our Solution
  7. Parameter $R$ in RNG Construction
  8. Multiple Parameter Recommendations
  9. Parameter Estimation Analysis
  10. Efficient Search Operations for Multiple PGs
  11. Efficient Prune Operations for Multiple PGs
  12. Efficient Construction of Multiple PGs
  13. Experiments
  14. Experimental Settings
  15. Main Results
  16. ...and 3 more sections

Key Result

Theorem 1

Given a data point $u\in D$ and a set $\{v_1,v_2,\cdots,v_L\}$ with $\delta(v_1,u)\le\delta(v_2,u)\le\cdots\le\delta(v_L,u)$, let $PN(R)$ be the results produced by Algorithm alg:prune with $C(u)=\{v_1,\cdots,v_R\}$. For the same $M$ and $\alpha$ values, it follows that $PN(R)\subseteq PN(R')$ when

Figures (9)

  • Figure 1: The effects of construction parameters on $k$-ANNS performance of PGs. QPS indicates the queries processed per second, while $Recall@k$ is the accuracy of $k$-ANN obtained.
  • Figure 2: The illustration of the PG construction methods.
  • Figure 3: Illustrating the framework of our method FastPGT.
  • Figure 4: The cost decomposition of PG construction.
  • Figure 5: The effects of $L$ and $\alpha$ on the neighbor list overlap between the Vamana graphs built accordingly on Sift.
  • ...and 4 more figures

Theorems & Definitions (4)

  • Theorem 1
  • proof
  • Theorem 2
  • proof