Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Passing the Baton: High Throughput Distributed Disk-Based Vector Search with BatANN

Nam Anh Dang, Ben Landrum, Ken Birman

TL;DR

BatANN addresses the challenge of scalable disk-based ANN on datasets that exceed single-node memory by distributing a global graph and exchanging full query states between servers. Its core innovation is asynchronous, state-passing beam search that preserves locality and minimizes inter-server round-trips, enabling near-linear throughput growth with additional servers while keeping latency low. Across 100M and 1B-scale benchmarks, BatANN significantly outperforms scatter-gather baselines and maintains efficient computation and I/O, demonstrating practical viability on standard TCP networks. The work provides open-source implementations and thoughtful design choices (head index, graph partitioning, and tuned I/O width) that advance high-throughput, low-latency distributed vector search on disk.

Abstract

Vector search underpins modern information-retrieval systems, including retrieval-augmented generation (RAG) pipelines and search engines over unstructured text and images. As datasets scale to billions of vectors, disk-based vector search has emerged as a practical solution. However, looking to the future, we need to anticipate datasets too large for any single server. We present BatANN, a distributed disk-based approximate nearest neighbor (ANN) system that retains the logarithmic search efficiency of a single global graph while achieving near-linear throughput scaling in the number of servers. Our core innovation is that when accessing a neighborhood which is stored on another machine, we send the full state of the query to the other machine to continue executing there for improved locality. On 100M- and 1B-point datasets at 0.95 recall using 10 servers, BatANN achieves 6.21-6.49x and 2.5-5.10x the throughput of the scatter-gather baseline, respectively, while maintaining mean latency below 6 ms. Moreover, we get these results on standard TCP. To our knowledge, BatANN is the first open-source distributed disk-based vector search system to operate over a single global graph.

Passing the Baton: High Throughput Distributed Disk-Based Vector Search with BatANN

TL;DR

BatANN addresses the challenge of scalable disk-based ANN on datasets that exceed single-node memory by distributing a global graph and exchanging full query states between servers. Its core innovation is asynchronous, state-passing beam search that preserves locality and minimizes inter-server round-trips, enabling near-linear throughput growth with additional servers while keeping latency low. Across 100M and 1B-scale benchmarks, BatANN significantly outperforms scatter-gather baselines and maintains efficient computation and I/O, demonstrating practical viability on standard TCP networks. The work provides open-source implementations and thoughtful design choices (head index, graph partitioning, and tuned I/O width) that advance high-throughput, low-latency distributed vector search on disk.

Abstract

Vector search underpins modern information-retrieval systems, including retrieval-augmented generation (RAG) pipelines and search engines over unstructured text and images. As datasets scale to billions of vectors, disk-based vector search has emerged as a practical solution. However, looking to the future, we need to anticipate datasets too large for any single server. We present BatANN, a distributed disk-based approximate nearest neighbor (ANN) system that retains the logarithmic search efficiency of a single global graph while achieving near-linear throughput scaling in the number of servers. Our core innovation is that when accessing a neighborhood which is stored on another machine, we send the full state of the query to the other machine to continue executing there for improved locality. On 100M- and 1B-point datasets at 0.95 recall using 10 servers, BatANN achieves 6.21-6.49x and 2.5-5.10x the throughput of the scatter-gather baseline, respectively, while maintaining mean latency below 6 ms. Moreover, we get these results on standard TCP. To our knowledge, BatANN is the first open-source distributed disk-based vector search system to operate over a single global graph.

Paper Structure

This paper contains 26 sections, 14 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Disk-based vector search
  4. Distributed Vector Search
  5. Scatter--Gather Approaches
  6. Distributed Global Indices
  7. The BatANN Approach
  8. What is a State?
  9. In-Memory Head Index
  10. Graph Partitioning
  11. I/O Pipeline Width
  12. Implementation and Optimizations
  13. Experiments
  14. Single Server
  15. Throughput
  16. ...and 11 more sections

Figures (14)

  • Figure 1: Scatter--Gather vs. Global Index
  • Figure 2: System overview
  • Figure 3: Number of hops vs. inter-partition hops for BIGANN 100M on 3, 5, 7, 10 servers with $W = 1$. Inter-partition hops only account for 11.6-24.3% of total hops, validating the choice to use graph partitioning.
  • Figure 4: Comparison of inter-partition hops between $W = 1, 8$ for BIGANN 1B on 10 servers. Higher $W$, fewer total and inter-partition hops.
  • Figure 5: Comparison of Distance Computations and Disk I/O issued between $W = 1, 8$ for BIGANN 1B on 10 server. Both $W$ values are nearly identical across all recall values.
  • ...and 9 more figures