KNN-DBSCAN: a DBSCAN in high dimensions

TL;DR

This article introduces a modified DBSCAN, using k-nearest neighbor (kNN) graphs to improve efficiency and outlines conditions for kNN-DBSCAN to match DBSCAN’s results and presents a parallel implementation using OpenMP and MPI for shared and distributed memory systems.

Abstract

Clustering is a fundamental task in machine learning. One of the most successful and broadly used algorithms is DBSCAN, a density-based clustering algorithm. DBSCAN requires $ε$-nearest neighbor graphs of the input dataset, which are computed with range-search algorithms and spatial data structures like KD-trees. Despite many efforts to design scalable implementations for DBSCAN, existing work is limited to low-dimensional datasets, as constructing $ε$-nearest neighbor graphs is expensive in high-dimensions. In this paper, we modify DBSCAN to enable use of $κ$-nearest neighbor graphs of the input dataset. The $κ$-nearest neighbor graphs are constructed using approximate algorithms based on randomized projections. Although these algorithms can become inaccurate or expensive in high-dimensions, they possess a much lower memory overhead than constructing $ε$-nearest neighbor graphs. We delineate the conditions under which $k$NN-DBSCAN produces the same clustering as DBSCAN. We also present an efficient parallel implementation of the overall algorithm using OpenMP for shared memory and MPI for distributed memory parallelism. We present results on up to 16 billion points in 20 dimensions, and perform weak and strong scaling studies using synthetic data. Our code is efficient in both low and high dimensions. We can cluster one billion points in 3D in less than one second on 28K cores on the Frontera system at the Texas Advanced Computing Center (TACC). In our largest run, we cluster 65 billion points in 20 dimensions in less than 40 seconds using 114,688 x86 cores on TACC's Frontera system. Also, we compare with a state of the art parallel DBSCAN code; on 20d/4M point dataset, our code is up to 37$\times$ faster.

Figures (12)

• Figure 1: Clustering results using K-Means, DBSCAN, and $k$NN-DBSCAN on 2D datasets. The second column shows the sorted $k$-distance plot for each dataset. Both DBSCAN and $k$NN-DBSCAN require two parameters: the range $\epsilon$ and the minimum number of points needed to form a dense region ($minPts$). For both algorithms, we set $minPts$ to the value of $k$ in the $k$-distance plots, with $\epsilon$ indicated by the red line in the plots. The $k$NN-DBSCAN algorithm produces identical clustering results to DBSCAN in these examples.
• Figure 2: Key definitions of DBSCAN: the hyperparameters are $\epsilon$ and $M=3$ in this example. Red dots are core points, green dot is border point, blue dots are noise points. $q$ is directly $\epsilon\mh$reachable from $p$ since $q\in \mathcal{N}_\epsilon(p)$; $q$ is not directly $M\mh$reachable from $p$ since $p\notin \mathcal{N}_{M}(q)$ and $q\notin \mathcal{N}_M(p)$. $q$ and $r$ are both directly $\epsilon\mh$reachable and directly $M\mh$reachable from each other.
• Figure 3: An illustration of the inexact MST. We partition points into three groups and consider $\epsilon\mh$radius neighborhood graph of these points, i.e., $G_\epsilon$. The inexact MST is a spanning tree combined by the local MST and cut MST. In this example, the inexact MST has three disconnected subtrees.
• Figure 4: The figures demonstrate the formation of cycles in the directed approximate $k$NN graph. In the top-left table, we have a directed approximate $k$NN graph with 4 points and $k=2$. In the bottom-left table, the directed graph is converted into an undirected graph by symmetrizing the edges. The minimum edges are highlighted in blue in both tables. The bottom-right graph shows the minimum edges from the undirected graph, which will not contain cycles once we break symmetry between $V_1$ and $V_2$. On the other hand, as depicted in the top-right graph, a cycle may exist in the directed graph, and pointer jumping will never end in such a graph.
• Figure 5: Illustration of distributed subtrees $\widehat{\mathcal{T}}$ and local roots $\widehat{\mathcal{T}}_l$ used in the first Boruvka's iteration of DistributedCutMST (see Algorithm \ref{['alg:cut_mst']}). We initialize 12 input local subtrees distributed in 4 MPI processes. After selecting minimum edges, subtrees are merged to form new ones. In the top row, old subtrees marked by the same color boxes belong to a new subtree now. We mark roots of newly formed subtrees with red numbers. In the last step of this iteration, $\widehat{\mathcal{T}}^0$ is updated to $\widehat{\mathcal{T}}^1$ by selecting all roots involved with each process. Root list $\widehat{\mathcal{T}}_l^1$ only consists of roots owned by each process.

Theorems & Definitions (16)

• Definition 3.1: classification of points
• Definition 3.2: direct $\epsilon\mh\text{reachable}$
• Definition 3.3: $\epsilon\mh\text{reachable}$
• Definition 3.4: $\epsilon\mh\text{connected}$
• Definition 3.5: cluster of DBSCAN
• lemma 1
• Definition 3.6: direct $M\mh\text{reachable}$
• Definition 3.7: $M\mh\text{reachable}$
• Definition 3.8: $M\mh\text{connected}$
• lemma 2