Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Scalable Overload-Aware Graph-Based Index Construction for 10-Billion-Scale Vector Similarity Search

Yang Shi, Yiping Sun, Jiaolong Du, Xiaocheng Zhong, Zhiyong Wang, Yao Hu

TL;DR

The paper tackles the challenge of constructing graph-based ANNS indexes at ultra-large scale, where traditional methods suffer from slow construction and limited scalability. It introduces SOGAIC, a scalable system that combines overload-aware adaptive data partitioning with a distributed load-balancing framework and an agglomerative subgraph merging strategy to assemble high-quality ANNS graphs across clusters. Empirical results show a 47.3\% average reduction in index construction time and robust performance on datasets up to $10^{10}$ points, with successful deployment in a real-world industrial search engine handling over $10^{10}$ daily vector updates. This work enables frequent updates and scalable vector similarity search in production environments with hundreds of millions of users.

Abstract

Approximate Nearest Neighbor Search (ANNS) is essential for modern data-driven applications that require efficient retrieval of top-k results from massive vector databases. Although existing graph-based ANNS algorithms achieve a high recall rate on billion-scale datasets, their slow construction speed and limited scalability hinder their applicability to large-scale industrial scenarios. In this paper, we introduce SOGAIC, the first Scalable Overload-Aware Graph-Based ANNS Index Construction system tailored for ultra-large-scale vector databases: 1) We propose a dynamic data partitioning algorithm with overload constraints that adaptively introduces overlaps among subsets; 2) To enable efficient distributed subgraph construction, we employ a load-balancing task scheduling framework combined with an agglomerative merging strategy; 3) Extensive experiments on various datasets demonstrate a reduction of 47.3% in average construction time compared to existing methods. The proposed method has also been successfully deployed in a real-world industrial search engine, managing over 10 billion daily updated vectors and serving hundreds of millions of users.

Scalable Overload-Aware Graph-Based Index Construction for 10-Billion-Scale Vector Similarity Search

TL;DR

The paper tackles the challenge of constructing graph-based ANNS indexes at ultra-large scale, where traditional methods suffer from slow construction and limited scalability. It introduces SOGAIC, a scalable system that combines overload-aware adaptive data partitioning with a distributed load-balancing framework and an agglomerative subgraph merging strategy to assemble high-quality ANNS graphs across clusters. Empirical results show a 47.3\% average reduction in index construction time and robust performance on datasets up to points, with successful deployment in a real-world industrial search engine handling over daily vector updates. This work enables frequent updates and scalable vector similarity search in production environments with hundreds of millions of users.

Abstract

Approximate Nearest Neighbor Search (ANNS) is essential for modern data-driven applications that require efficient retrieval of top-k results from massive vector databases. Although existing graph-based ANNS algorithms achieve a high recall rate on billion-scale datasets, their slow construction speed and limited scalability hinder their applicability to large-scale industrial scenarios. In this paper, we introduce SOGAIC, the first Scalable Overload-Aware Graph-Based ANNS Index Construction system tailored for ultra-large-scale vector databases: 1) We propose a dynamic data partitioning algorithm with overload constraints that adaptively introduces overlaps among subsets; 2) To enable efficient distributed subgraph construction, we employ a load-balancing task scheduling framework combined with an agglomerative merging strategy; 3) Extensive experiments on various datasets demonstrate a reduction of 47.3% in average construction time compared to existing methods. The proposed method has also been successfully deployed in a real-world industrial search engine, managing over 10 billion daily updated vectors and serving hundreds of millions of users.

Paper Structure

This paper contains 10 sections, 3 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. System Overview
  3. Overload-Aware Adaptive Data Partitioning
  4. Distributed ANNS Graph Construction with Agglomerative Subgraph Merging
  5. Experiment
  6. Experimental Setting
  7. Experimental Result and Analysis
  8. Construction Performance
  9. System Scalability
  10. Conclusion

Figures (3)

  • Figure 1: (a) Illustration of the assignment process for P1 and P2 based on their geometric relationships with centroids {C1 ... C4}. (b) Illustration of the deviation from P to P' for the assignment process of P, when overload happens on division1. (c) Illustration of the overall data partitioning process for P with quantization encoding executed in parallel.
  • Figure 2: (a) Parallel build and merge framework versus Sequential one. (b) Agglomerative subgraph merging topology.
  • Figure 3: Comparisons of the proposed method (SOGAIC) with others (Faiss HNSW, DiskANN, SPTAG) on both performance and scalability for different datasets: SIFT1M, SIFT1B, GloVe, ISD3B, VDD10B.