DET-LSH: A Locality-Sensitive Hashing Scheme with Dynamic Encoding Tree for Approximate Nearest Neighbor Search

TL;DR

DET-LSH introduces a novel encoding-based indexing tree (DE-Tree) and a locality-sensitive hashing scheme that leverages dynamic per-dimension encoding (via iSAX-like representations) to enable efficient range queries and accurate approximate nearest neighbor search in high-dimensional spaces. By constructing multiple independent DE-Trees and combining coarse range filtering with fine distance calculations, the approach achieves a $c^2$-$k$-ANN with constant probability while significantly reducing indexing time and maintaining high recall. Theoretical guarantees are complemented by extensive experiments showing up to 6× indexing speedup and 2× query speedup over state-of-the-art LSH methods, and favorable scalability compared to graph-based methods. DET-LSH thereby offers a practical, scalable solution for large-scale ANN in real-world datasets with strong accuracy guarantees.

Abstract

Locality-sensitive hashing (LSH) is a well-known solution for approximate nearest neighbor (ANN) search in high-dimensional spaces due to its robust theoretical guarantee on query accuracy. Traditional LSH-based methods mainly focus on improving the efficiency and accuracy of the query phase by designing different query strategies, but pay little attention to improving the efficiency of the indexing phase. They typically fine-tune existing data-oriented partitioning trees to index data points and support their query strategies. However, their strategy to directly partition the multi-dimensional space is time-consuming, and performance degrades as the space dimensionality increases. In this paper, we design an encoding-based tree called Dynamic Encoding Tree (DE-Tree) to improve the indexing efficiency and support efficient range queries based on Euclidean distance. Based on DE-Tree, we propose a novel LSH scheme called DET-LSH. DET-LSH adopts a novel query strategy, which performs range queries in multiple independent index DE-Trees to reduce the probability of missing exact NN points, thereby improving the query accuracy. Our theoretical studies show that DET-LSH enjoys probabilistic guarantees on query accuracy. Extensive experiments on real-world datasets demonstrate the superiority of DET-LSH over the state-of-the-art LSH-based methods on both efficiency and accuracy. While achieving better query accuracy than competitors, DET-LSH achieves up to 6x speedup in indexing time and 2x speedup in query time over the state-of-the-art LSH-based methods. This paper was published in PVLDB 2024.

Figures (12)

• Figure 1: Illustration of an encoding-based tree.
• Figure 2: Overview of the DET-LSH workflow.
• Figure 3: Illustration of the theoretical $\beta$ when $L$ varies (for $K=16$ and $c=1.5$), which is in line with Lemma \ref{['lemma3']}.
• Figure 4: Running time break-down for the DET-LSH encoding and indexing phases.
• Figure 5: Running time and recall of optimized/non-optimized query-phase algorithms of DET-LSH.

Theorems & Definitions (9)

• Definition 1: $c$-ANN
• Definition 2: $c$-$k$-ANN
• Definition 3: ($r$,$c$)-ANN
• Definition 4: LSH
• Lemma 1
• Lemma 2
• Lemma 3
• Theorem 1
• Theorem 2