STABLE: Efficient Hybrid Nearest Neighbor Search via Magnitude-Uniformity and Cardinality-Robustness

Abstract

Hybrid Approximate Nearest Neighbor Search (Hybrid ANNS) is a foundational search technology for large-scale heterogeneous data and has gained significant attention in both academia and industry. However, current approaches overlook the heterogeneity in data distribution, thus ignoring two major challenges: the Compatibility Barrier for Similarity Magnitude Heterogeneity and the Tolerance Bottleneck to Attribute Cardinality. To overcome these issues, we propose the robuSt heTerogeneity-Aware hyBrid retrievaL framEwork, STABLE, designed for accurate, efficient, and robust hybrid ANNS under datasets with various distributions. Specifically, we introduce an enhAnced heterogeneoUs semanTic perceptiOn (AUTO) metric to achieve a joint measurement of feature similarity and attribute consistency, addressing similarity magnitude heterogeneity and improving robustness to datasets with various attribute cardinalities. Thereafter, we construct our Heterogeneous sEmantic reLation graPh (HELP) index based on AUTO to organize heterogeneous semantic relations. Finally, we employ a novel Dynamic Heterogeneity Routing method to ensure an efficient search. Extensive experiments on five feature vector benchmarks with various attribute cardinalities demonstrate the superior performance of STABLE.

Figures (10)

• Figure 1: Illustrations of (a) ANNS, (b) Hybrid ANNS, and (c) performance comparison under various attribute cardinalities.
• Figure 2: The iteration of HELP construction. The figure illustrates the neighbor update of $D_1$, with the blue node indicating the other nodes within $\mathcal{D}$.
• Figure 3: (a)-(i) represent the QPS vs Recall@$10$ performance of hybrid ANNS methods on nine 1M-scale datasets.
• Figure 4: (a)-(f) represent the QPS vs Recall@$10$ performance of hybrid ANNS methods on six 10M-scale datasets.
• Figure 5: Performance comparison under varying query selectivities on SIFT-7-3 dataset. The subplots display QPS vs. Recall@10 as the number of active filters ($F$) increases from 1 to 7. The percentage in parentheses indicates the approximate selectivity.

Theorems & Definitions (2)

• Remark 1
• Remark 2