How Should We Evaluate Data Deletion in Graph-Based ANN Indexes?

TL;DR

This work addresses the challenge of evaluating and managing data deletion in graph-based ANNS under dynamic updates. It formalizes three deletion paradigms (logical, physical, rebuilding), integrates them into HNSW, and establishes a deployment-oriented evaluation protocol with metrics like 1-Recall@10 and QPS. The authors also introduce Deletion Control, an adaptive strategy that estimates dataset-dependent thresholds to switch deletion methods while maintaining target accuracy. Collectively, the paper provides methodological guidance and practical mechanisms for balancing deletion throughput, memory usage, and search performance in real-world dynamic ANN systems.

Abstract

Approximate Nearest Neighbor Search (ANNS) has recently gained significant attention due to its many applications, such as Retrieval-Augmented Generation. Such applications require ANNS algorithms that support dynamic data, so the ANNS problem on dynamic data has attracted considerable interest. However, a comprehensive evaluation methodology for data deletion in ANNS has yet to be established. This study proposes an experimental framework and comprehensive evaluation metrics to assess the efficiency of data deletion for ANNS indexes under practical use cases. Specifically, we categorize data deletion methods in graph-based ANNS into three approaches and formalize them mathematically. The performance is assessed in terms of accuracy, query speed, and other relevant metrics. Finally, we apply the proposed evaluation framework to Hierarchical Navigable Small World, one of the state-of-the-art ANNS methods, to analyze the effects of data deletion, and propose Deletion Control, a method which dynamically selects the appropriate deletion method under a required search accuracy.

Figures (14)

• Figure 1: Overview of three data deletion methods: logical deletion, physical deletion, and rebuilding.
• Figure 1: Experimental conditions for each dataset
• Figure 2: Evaluation of deletion performance on SIFT1M and SIFT1B datasets. (a) Comparison of search performance after step 5 deletion in SIFT1M. (b, c) QPS for deletion on SIFT1B with varying batch sizes. (d) Impact on 1-Recall@10 after updates on SIFT1B.
• Figure 3: Comparison of 1-Recall@10 on SIFT1B training set (10$\%$ queries).
• Figure 4: Comparison of 1-Recall@10 on test set (all queries).