Enhancing HNSW Index for Real-Time Updates: Addressing Unreachable Points and Performance Degradation

TL;DR

The paper tackles the deterioration of HNSW performance under real-time deletions/insertions, notably the unreachable points phenomenon and degraded update efficiency. It introduces MN-RU, a mutual-neighbor replaced-update scheme, together with a HNSW Backup Index and a dualSearch mechanism to preserve graph connectivity and retrieval accuracy during dynamic updates. Across multiple datasets, the approach reduces unreachable point growth and achieves 2–4x faster update times, while maintaining recall for top-$k$ queries; the backup index further stabilizes accuracy under churn. The methods are designed to integrate with existing HNSW-based systems, offering practical applicability for real-world, high-update environments and opening avenues for robust dynamic graph-based ANNS research.

Abstract

The approximate nearest neighbor search (ANNS) is a fundamental and essential component in data mining and information retrieval, with graph-based methodologies demonstrating superior performance compared to alternative approaches. Extensive research efforts have been dedicated to improving search efficiency by developing various graph-based indices, such as HNSW (Hierarchical Navigable Small World). However, the performance of HNSW and most graph-based indices become unacceptable when faced with a large number of real-time deletions, insertions, and updates. Furthermore, during update operations, HNSW can result in some data points becoming unreachable, a situation we refer to as the `unreachable points phenomenon'. This phenomenon could significantly affect the search accuracy of the graph in certain situations. To address these issues, we present efficient measures to overcome the shortcomings of HNSW, specifically addressing poor performance over long periods of delete and update operations and resolving the issues caused by the unreachable points phenomenon. Our proposed MN-RU algorithm effectively improves update efficiency and suppresses the growth rate of unreachable points, ensuring better overall performance and maintaining the integrity of the graph. Our results demonstrate that our methods outperform existing approaches. Furthermore, since our methods are based on HNSW, they can be easily integrated with existing indices widely used in the industrial field, making them practical for future real-world applications. Code is available at \url{https://github.com/xwt1/MN-RU.git}

Figures (27)

• Figure 1: Figure 1: Comparison of query efficiency and replaced_update efficiency at a given recall level on three public datasets.
• Figure 2: Figure 2: Example Of Unreachable Points Phenomenon
• Figure 4: Figure 4: The architecture of our work, both in upper-level application and MN-RU
• Figure 5: Figure 5: Example of repairing neighbors of points u and v after deleting point d.
• Figure 10: Figure 10: Search performance following update operations in full_coverage scenario using the Gist and ImageNet dataset.

Theorems & Definitions (1)

• Definition 1