A Memory-Efficient Distributed Algorithm for Approximate Nearest Neighbour Search with Arbitrary Distances

TL;DR

The paper tackling ANN in heterogeneous, high-dimensional data with memory constraints introduces PDASC, a distributed, distance-agnostic ANN built on a hierarchical index created via multilevel clustering. PDASC supports arbitrary distance functions and distributes storage and search across nodes, mitigating the curse of dimensionality through dense local representations. It demonstrates competitive recall with lower per-node memory than state-of-the-art methods and operates without specialized hardware. The work enables scalable, energy-efficient similarity search in distributed, resource-constrained settings, and outlines avenues for further optimization and language/runtime improvements.

Abstract

Approximate nearest neighbour (ANN) search has become a central task in modern data-intensive applications, particularly when operating on large, heterogeneous, or high-dimensional datasets. However, many existing ANN methods struggle in such scenarios, either because they rely on metric assumptions or because their indexing strategies are not well suited to distributed environments or to settings with constrained memory resources. This work introduces PDASC (Parametrizable Distributed Approximate Similarity Search with Clustering), a distributed ANN search algorithm whose index design simultaneously supports arbitrary dissimilarity functions and efficient deployment in distributed, storage-aware environments. PDASC builds a distributed hierarchical index based on clustering mechanisms that are agnostic to distance properties, thereby accommodating non-metric and domain-specific similarities while naturally partitioning indexing and search across multiple computing nodes, with a compact per-node memory footprint. By preserving locally informative neighbourhood structure, the proposed index mitigates practical manifestations of the curse of dimensionality in high-dimensional spaces. We analyse how the index structural parameters govern the trade-offs among recall, computational cost, and memory usage. Experimental evaluation across multiple benchmark datasets and distance functions shows that PDASC achieves competitive accuracy-efficiency trade-offs while consistently requiring lower per-node memory compared to state-of-the-art ANN methods. By avoiding reliance on specialised hardware acceleration, PDASC enables scalable and energy-efficient similarity search in heterogeneous and distributed settings where memory efficiency and distance-function flexibility are first-class constraints.

Figures (5)

• Figure 1: Conceptual representation of the multilevel index construction in PDASC using the MSA.
• Figure 2: Conceptual illustration of the $k$-NN search in PDASC using the NSA over a multilevel index distributed across $nNodes = 3$ computing nodes. The dataset of 65 elements is partitioned into subsets of 24 (node 0), 24 (node 1), and 11 elements (node 2), where each local index is built using $gl = 6$ and $np = 3$.
• Figure 3: PDASC Performance on GLOVE dataset under different degrees of parallelisation ($nNodes$).
• Figure 4: Per-node index size versus recall for PDASC and the evaluated ANN methods across all datasets (x-axis in logarithmic scale).
• Figure 5: Recall versus NDC for PDASC and IVF on the NYTimes dataset.