Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MCGI: Manifold-Consistent Graph Indexing for Billion-Scale Disk-Resident Vector Search

Dongfang Zhao

TL;DR

MCGI tackles the challenge of high-dimensional disk-resident vector search by tying graph routing to the intrinsic geometry of data via Local Intrinsic Dimensionality (LID). It introduces a two-phase, manifold-aware indexing framework with a mapping function that adaptively prunes edges based on local geometry, yielding theoretical guarantees on connectivity and adaptive routing complexity. Empirically, MCGI achieves state-of-the-art throughput and latency reductions on GIST1M and SIFT1B, while preserving performance on lower-dimensional benchmarks, effectively narrowing the gap between disk-based and in-memory ANN systems. The work demonstrates that incorporating manifold-awareness into indexing enables scalable, efficient billion-scale vector search with practical impact for retrieval-augmented systems and similar applications.

Abstract

Graph-based Approximate Nearest Neighbor (ANN) search often suffers from performance degradation in high-dimensional spaces due to the ``Euclidean-Geodesic mismatch,'' where greedy routing diverges from the underlying data manifold. To address this, we propose Manifold-Consistent Graph Indexing (MCGI), a geometry-aware and disk-resident indexing method that leverages Local Intrinsic Dimensionality (LID) to dynamically adapt search strategies to the data's intrinsic geometry. Unlike standard algorithms that treat dimensions uniformly, MCGI modulates its beam search budget based on in situ geometric analysis, eliminating dependency on static hyperparameters. Theoretical analysis confirms that MCGI enables improved approximation guarantees by preserving manifold-consistent topological connectivity. Empirically, MCGI achieves 5.8$\times$ higher throughput at 95\% recall on high-dimensional GIST1M compared to state-of-the-art DiskANN. On the billion-scale SIFT1B dataset, MCGI further validates its scalability by reducing high-recall query latency by 3$\times$, while maintaining performance parity on standard lower-dimensional datasets.

MCGI: Manifold-Consistent Graph Indexing for Billion-Scale Disk-Resident Vector Search

TL;DR

MCGI tackles the challenge of high-dimensional disk-resident vector search by tying graph routing to the intrinsic geometry of data via Local Intrinsic Dimensionality (LID). It introduces a two-phase, manifold-aware indexing framework with a mapping function that adaptively prunes edges based on local geometry, yielding theoretical guarantees on connectivity and adaptive routing complexity. Empirically, MCGI achieves state-of-the-art throughput and latency reductions on GIST1M and SIFT1B, while preserving performance on lower-dimensional benchmarks, effectively narrowing the gap between disk-based and in-memory ANN systems. The work demonstrates that incorporating manifold-awareness into indexing enables scalable, efficient billion-scale vector search with practical impact for retrieval-augmented systems and similar applications.

Abstract

Graph-based Approximate Nearest Neighbor (ANN) search often suffers from performance degradation in high-dimensional spaces due to the ``Euclidean-Geodesic mismatch,'' where greedy routing diverges from the underlying data manifold. To address this, we propose Manifold-Consistent Graph Indexing (MCGI), a geometry-aware and disk-resident indexing method that leverages Local Intrinsic Dimensionality (LID) to dynamically adapt search strategies to the data's intrinsic geometry. Unlike standard algorithms that treat dimensions uniformly, MCGI modulates its beam search budget based on in situ geometric analysis, eliminating dependency on static hyperparameters. Theoretical analysis confirms that MCGI enables improved approximation guarantees by preserving manifold-consistent topological connectivity. Empirically, MCGI achieves 5.8 higher throughput at 95\% recall on high-dimensional GIST1M compared to state-of-the-art DiskANN. On the billion-scale SIFT1B dataset, MCGI further validates its scalability by reducing high-recall query latency by 3, while maintaining performance parity on standard lower-dimensional datasets.
Paper Structure (32 sections, 5 theorems, 13 equations, 3 figures, 3 tables, 1 algorithm)

This paper contains 32 sections, 5 theorems, 13 equations, 3 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Vector Indexing Paradigms.
  4. High-Dimensional Indexing.
  5. Intrinsic Dimensionality.
  6. Methodology
  7. Definitions
  8. Mapping Function
  9. Manifold-Consistent Graph Indexing
  10. Phase 1: Geometric Calibration.
  11. Phase 2: Manifold-Consistent Refinement.
  12. Theoretical Analysis
  13. Geometric Complexity and Adaptive Routing
  14. Topological Fidelity and Connectivity
  15. System Implementation
  16. ...and 17 more sections

Key Result

Proposition 3.4

The mapping function $\Phi$ is strictly decreasing with respect to the estimated local intrinsic dimensionality. Formally, given that the standard deviation of the LID estimates $\sigma_{\widehat{\text{LID}}} > 0$ and the pruning range $\alpha_{\max} > \alpha_{\min}$, the derivative satisfies:

Figures (3)

  • Figure 1: Recall-QPS Trade-off. Comparison of MCGI against DiskANN and Faiss (mmap) on three datasets.
  • Figure 2: Billion-Scale Performance on SIFT1B.
  • Figure 3: Resource Efficiency (RQ4).

Theorems & Definitions (8)

  • Definition 3.1: Local Intrinsic Dimensionality
  • Remark 3.2: Institution of LID
  • Definition 3.3: LID Maximum Likelihood Estimator
  • Proposition 3.4: Monotonicity
  • Proposition 3.5: Boundedness
  • Lemma 4.1: Local Complexity Lower Bound
  • Proposition 4.2: Optimal Budget Allocation
  • Proposition 4.3: Connectivity Preservation