ParlayANN: Scalable and Deterministic Parallel Graph-Based Approximate Nearest Neighbor Search Algorithms

TL;DR

ParlayANN delivers a deterministic, scalable framework for graph-based ANNS that scales to billions of points on a single machine by introducing prefix-doubling, batch insertion, and lock-free merging across four state-of-the-art graphs (DiskANN, HNSW, HCNNG, PyNNDescent). The approach achieves high recall (often near 1.0 for 10@10) with substantial parallel throughput (QPS in the 10^4 range) and builds indexes in about 10 hours on large hardware, outperforming or matching non-graph baselines at billion-scale across diverse datasets and OOD queries. The study provides extensive cross-algorithm benchmarking, reveals insights into scalability bottlenecks, and highlights HCNNG as a particularly strong yet under-exploited approach. Overall, ParlayANN demonstrates that deterministic parallel graph-based ANNS can rival and exceed existing methods on very large-scale vector search tasks with practical build and query performance.

Abstract

Approximate nearest-neighbor search (ANNS) algorithms are a key part of the modern deep learning stack due to enabling efficient similarity search over high-dimensional vector space representations (i.e., embeddings) of data. Among various ANNS algorithms, graph-based algorithms are known to achieve the best throughput-recall tradeoffs. Despite the large scale of modern ANNS datasets, existing parallel graph based implementations suffer from significant challenges to scale to large datasets due to heavy use of locks and other sequential bottlenecks, which 1) prevents them from efficiently scaling to a large number of processors, and 2) results in nondeterminism that is undesirable in certain applications. In this paper, we introduce ParlayANN, a library of deterministic and parallel graph-based approximate nearest neighbor search algorithms, along with a set of useful tools for developing such algorithms. In this library, we develop novel parallel implementations for four state-of-the-art graph-based ANNS algorithms that scale to billion-scale datasets. Our algorithms are deterministic and achieve high scalability across a diverse set of challenging datasets. In addition to the new algorithmic ideas, we also conduct a detailed experimental study of our new algorithms as well as two existing non-graph approaches. Our experimental results both validate the effectiveness of our new techniques, and lead to a comprehensive comparison among ANNS algorithms on large scale datasets with a list of interesting findings.

Figures (8)

• Figure 1: Scalability of original and our new implementations of four ANNS algorithms on various number of threads. Within each subfigure, all numbers are speedup numbers relative to the original implementation on one thread. Higher is better. Results were tested on a machine with 48 cores using dataset BIGANN-1M ($10^6$ points). "48h": 48 cores with hyperthreads. The two implementations in the same subfigure always use the same parameters and give similar query quality (recall-QPS curve).
• Figure 2: An example of ANNS graph and a greedy search. The blue arrows represent directed edges in the proximity graph, which is a mix of long and short edges. Below is an example of NNS query on point $q$ (red point). The algorithm starts with adding the starting point $A$ as the only point in the beam $\mathcal{L}$, and then in every step, finds the closest unprocessed point in $\mathcal{L}$ (to $q$) and adds its out-neighbors. Once $|\mathcal{L}|$ goes beyond $L$, it is refined to keep only the $L$ nearest points. A set $\mathcal{V}$ is maintained for all processed vertices. When all vertices in $\mathcal{L}$ are also in $\mathcal{V}$, the algorithm finishes.
• Figure 3: Build time (hours), QPS, recall, and distance comparisons for all algorithms on billion-size datasets.
• Figure 4: QPS-recall curves on all 100-million size datasets. The first row shows the overall QPS/recall curve, while the second row zooms into a higher-recall regime. The build times are given in \ref{['fig:buildtimeshundredmil']}
• Figure 5: QPS on a single thread on BIGANN-1M. Shown to compare with ANN-benchmarks.

Theorems & Definitions (2)

• definition 1
• definition 2