Graph-Based Algorithms for Diverse Similarity Search

TL;DR

This work introduces provably efficient graph-based algorithms for approximate nearest neighbor search under diversity constraints, addressing the inefficiency of traditional two-stage reranking. By extending DiskANN to a color-aware framework, it achieves diversity-aware NN with guarantees that scale with the data's doubling dimension $d$ and aspect ratio $\Delta$, yielding query time $O\left(k\log_{\alpha}\frac{\Delta}{\epsilon}\right)$ steps and per-step time $\tilde{O}\left(k(8\alpha)^d\log \Delta\right)$, while using space $O\left(nk(8\alpha)^d\log \Delta\right)$. The framework generalizes to arbitrary $k'$-colorful constraints and generic diversity metrics $\rho$, enabling primal and dual formulations and preserving approximation guarantees up to factors depending on $\alpha$ and $\epsilon$. Experimental results on real-world and semi-synthetic datasets show substantial latency reductions and recall improvements compared to the baseline two-stage approach, with practical implications for online advertising, recommendations, and information retrieval where diversity of results is critical.

Abstract

Nearest neighbor search is a fundamental data structure problem with many applications in machine learning, computer vision, recommendation systems and other fields. Although the main objective of the data structure is to quickly report data points that are closest to a given query, it has long been noted (Carbonell and Goldstein, 1998) that without additional constraints the reported answers can be redundant and/or duplicative. This issue is typically addressed in two stages: in the first stage, the algorithm retrieves a (large) number $r$ of points closest to the query, while in the second stage, the $r$ points are post-processed and a small subset is selected to maximize the desired diversity objective. Although popular, this method suffers from a fundamental efficiency bottleneck, as the set of points retrieved in the first stage often needs to be much larger than the final output. In this paper we present provably efficient algorithms for approximate nearest neighbor search with diversity constraints that bypass this two stage process. Our algorithms are based on popular graph-based methods, which allows us to "piggy-back" on the existing efficient implementations. These are the first graph-based algorithms for nearest neighbor search with diversity constraints. For data sets with low intrinsic dimension, our data structures report a diverse set of $k$ points approximately closest to the query, in time that only depends on $k$ and $\log Δ$, where $Δ$ is the ratio of the diameter to the closest pair distance in the data set. This bound is qualitatively similar to the best known bounds for standard (non-diverse) graph-based algorithms. Our experiments show that the search time of our algorithms is substantially lower than that using the standard two-stage approach.

Figures (5)

• Figure 1: Seller distribution in a real-world dataset with 20 million base vectors, where the top 7 sellers constitute more than 90% of the data.
• Figure 2: Brand cumulative distribution for Amazon dataset, showing the coverage of the vectors by the brands in sorted order. The top $10\%$ of brands cover $86\%$ of the vectors.
• Figure 3: Recall vs Latency for $k'=1$.
• Figure 4: Recall vs Latency for $k'=10$.
• Figure 5: Recall vs Latency for SIFT-Skewed dataset with $k'=10$ (left) and $k'=1$ (right) by varying the diversity parameter $m$ during index construction. Higher $m$ implies more diversity.

Theorems & Definitions (32)

• Theorem 1.1
• Definition 2.1: colorful
• Definition 2.2: $k'$-colorful
• Definition 2.3
• Definition 2.4: Colorful NN
• Definition 2.5: $k'$-Colorful NN
• Lemma 2.6
• Lemma 3.1
• proof
• Proposition 3.2