Accelerating High-Dimensional Nearest Neighbor Search with Dynamic Query Preference
Yifan Zhu, Ruijie Zhao, Zhonggen Li, Baihua Zheng, Zhikun Zhang, Zhaoqiang Chen, Congcong Ge
TL;DR
This paper tackles the challenge of high-dimensional approximate nearest neighbor search under skewed and time-varying query workloads. It introduces the Dual-Index Query Framework (DQF), which separates hot high-frequency items into a compact hot index while retaining a full index for the entire dataset, and couples this with a dynamic, decision-tree–guided search to terminate queries early when possible. The main contributions include a NSSG-based construction for both hot and full indexes, a hot-index update mechanism that incrementally adapts to shifting query patterns, and a dynamic search strategy that significantly speeds up queries while preserving recall. Experiments on four real-world datasets show 2.0–5.7× speedups with about 95% recall and demonstrate that the approach remains effective under distribution shifts without requiring full index rebuilds, highlighting practical applicability for systems with evolving user preferences.
Abstract
Approximate Nearest Neighbor Search (ANNS) is a crucial operation in databases and artificial intelligence. Current graph-based ANNS methods, such as HNSW and NSG, have shown remarkable performance but are designed under the assumption of a uniform query distribution. However, in practical scenarios, user preferences and query temporal dynamics lead to some queries being searched for more frequently than others. To fully utilize these characteristics, we propose DQF, a novel Dual-Index Query Framework. This framework comprises a dual-layer index structure and a dynamic search strategy based on a decision tree. The dual-layer index structure comprises a hot index for high-frequency nodes and a full index for the entire dataset, allowing for the separate management of hot and cold queries. Furthermore, we propose a dynamic search strategy that employs a decision tree to adapt to the specific characteristics of each query. The decision tree evaluates whether a query is of the high-frequency type to detect the opportunities for early termination on the dual-layer, avoiding unnecessary searches in the full index. Experimental results on four real-world datasets demonstrate that the Dual-Index Query Framework achieves a significant speedup of 2.0-5.7x over state-of-the-art algorithms while maintaining a 95% recall rate. Importantly, it does not require full index reconstruction when query distributions change, underscoring its efficiency and practicality in dynamic query distribution scenarios.