A new near-linear time algorithm for k-nearest neighbor search using a compressed cover tree

TL;DR

The paper addresses the exact $k$-nearest neighbor problem in metric spaces and introduces a compressed cover tree $\mathcal{T}(R)$ to achieve near-linear time in $\max\{|Q|,|R|\}$. It presents a rigorous construction procedure with time bounds $O((c_m(R))^{8}\cdot \log_2(\Delta(R))\cdot |R|)$ (and $O((c(R))^{12}\cdot \log_2|R|\cdot |R|)$ under alternative constants) and a refined $k$-NN search algorithm that uses a $\lambda$-point and a new break condition to bound iterations, yielding per-query bounds like $O\left((c_m(R))^{10}\cdot \log_2(k)\cdot \log_2(\Delta(R)) + |\bar{B}(q,5d_k(q,R))|\cdot \log_2(k)\right)$ and a fixed-constant version $O\left(\log_2(k)\cdot(\log_2|R|+k)\right)$. The results remove gaps in prior analyses, enabling rigorous near-linear performance for exact $k$-NN on large-scale datasets and informing applications in geometric data science and materials informatics.

Abstract

Given a reference set $R$ of $n$ points and a query set $Q$ of $m$ points in a metric space, this paper studies an important problem of finding $k$-nearest neighbors of every point $q \in Q$ in the set $R$ in a near-linear time. In the paper at ICML 2006, Beygelzimer, Kakade, and Langford introduced a cover tree on $R$ and attempted to prove that this tree can be built in $O(n\log n)$ time while the nearest neighbor search can be done in $O(n\log m)$ time with a hidden dimensionality factor. This paper fills a substantial gap in the past proofs of time complexity by defining a simpler compressed cover tree on the reference set $R$. The first new algorithm constructs a compressed cover tree in $O(n \log n)$ time. The second new algorithm finds all $k$-nearest neighbors of all points from $Q$ using a compressed cover tree in time $O(m(k+\log n)\log k)$ with a hidden dimensionality factor depending on point distributions of the given sets $R,Q$ but not on their sizes.

Figures (13)

• Figure 1: A comparison of past cover trees and a new compressed cover tree in Example \ref{['exa:implicitexplicitexample']}. Left: an implicit cover tree contains infinite repetitions. Middle: an explicit cover tree. Right: a compressed cover tree from Definition \ref{['dfn:cover_tree_compressed']} includes each given point exactly once.
• Figure 2: Consider a compressed cover tree $\mathcal{T}(R)$ that was built on set $R = \{1,2,3,4,5,7,8\}$. Let $\mathcal{S}_i(p, \mathcal{T}(R))$ be a distinctive descendant set of Definition \ref{['dfn:distinctive_descendant_set']}. Then $V_2(1) = \emptyset, V_{1}(1) = \{5\}$ and $V_{0}(1) = \{3,5,7\}$. And also $\mathcal{S}_2(1, \mathcal{T}(R)) = \{1, 2,3,4,5,7,8\}$, $\mathcal{S}_1(1, \mathcal{T}(R)) = \{1,2,3,4\}$ and $\mathcal{S}_{0}(1, \mathcal{T}(R)) = \{1\}$.
• Figure 3: Left: an implicit cover tree from beygelzimer2006cover at ICML 2006 for a finite set of reference points $R = \{1,2,3,4,5\}$ with the Euclidean distance $d(x,y) = |x-y|$. Right: a new compressed cover tree in Definition \ref{['dfn:cover_tree_compressed']} corrects the past worst-case complexity for $k$-nearest neighbors search in $R$.
• Figure 4: Compressed cover trees $\mathcal{T}(R)$ from Definition \ref{['dfn:cover_tree_compressed']} for $R = \{0,1,2^{i}\}$.
• Figure 5: Compressed cover tree $\mathcal{T}(R)$ on the set $R$ in Example \ref{['exa:cover_tree_big']} with root $16$.

Theorems & Definitions (163)

• Definition 1.0: diameter and aspect ratio
• Definition 1.0: $k$-nearest neighbor set $\NN_k$
• Definition 1.0: expansion constants $c$ and $c_m$
• Lemma 1.0
• Definition 2.0: a compressed cover tree $\T(R)$
• Lemma 2.0: packing
• Lemma 2.0: width bound
• Lemma 2.0: growth bound
• Lemma 2.0: extended growth bound
• Definition 2.0: the height of a compressed cover tree