Efficient Graph-Based Approximate Nearest Neighbor Search Achieving: Low Latency Without Throughput Loss

TL;DR

High-dimensional embeddings drive better semantic similarity but raise ANNS latency and throughput challenges in real-time applications. AverSearch introduces a fully asynchronous graph-based ANNS framework with dynamic workload balancing and a three-role threading model to minimize synchronization and redundant work. The approach yields up to 2.1–8.9x higher throughput at comparable latency and reduces minimum latency by 1.5–1.9x, while linking throughput to memory bandwidth via $EMB = PMB imes (1 - RR)$. This has practical impact for retrieval-intensive tasks such as RAG and long-context attention, enabling faster, more scalable, real-time vector search on CPU hardware.

Abstract

The increase in the dimensionality of neural embedding models has enhanced the accuracy of semantic search capabilities but also amplified the computational demands for Approximate Nearest Neighbor Searches (ANNS). This complexity poses significant challenges in online and interactive services, where query latency is a critical performance metric. Traditional graph-based ANNS methods, while effective for managing large datasets, often experience substantial throughput reductions when scaled for intra-query parallelism to minimize latency. This reduction is largely due to inherent inefficiencies in the conventional fork-join parallelism model. To address this problem, we introduce AverSearch, a novel parallel graph-based ANNS framework that overcomes these limitations through a fully asynchronous architecture. Unlike existing frameworks that struggle with balancing latency and throughput, AverSearch utilizes a dynamic workload balancing mechanism that supports continuous, dependency-free processing. This approach not only minimizes latency by eliminating unnecessary synchronization and redundant vertex processing but also maintains high throughput levels. Our evaluations across various datasets, including both traditional benchmarks and modern large-scale model generated datasets, show that AverSearch consistently outperforms current state-of-the-art systems. It achieves up to 2.1-8.9 times higher throughput at comparable latency levels across different datasets and reduces minimum latency by 1.5 to 1.9 times.

Figures (11)

• Figure 1: The relationship between latency and throughput, expressed in queries per second (QPS), varies across different parallelism settings. We leverage all 48 available cores, organizing them into "$intra \times inter$" groups, ranging from "1 $\times$ 48" to "24 $\times$ 2". Here, "$intra$" represents the number of threads dedicated to each query, while "$inter$" indicates the number of independent concurrent queries. The analysis uses two well-known datasets, SIFT100M and Wiki100M wikipedia-cohere-2022, each containing 100 million vectors with dimensions of 128 and 768, respectively. The evaluation covers various recall levels from 0.9 to 0.995.
• Figure 2: iQAN execution time distribution under different parallelism strategies with Wiki100M and recall at 0.9.
• Figure 3: Edge-wise parallelism v.s. path-wise parallelism (§\ref{['sec:pbfis']})
• Figure 4: iQAN CPU time breakdown for a single query with varying accuracy and parallelism strategies. "Serial time" for the joining phase; "Expand time" for useful expansions during the forking phase; "Redundant time" for unnecessary expansions; and "Sync time" resulting from thread synchronization and load imbalance.
• Figure 5: iQAN CPU time breakdown for a single query across varying $width$ with 32 threads.