PilotANN: Memory-Bounded GPU Acceleration for Vector Search

TL;DR

PilotANN addresses the memory bottleneck of graph-based ANNS on large-scale, high-dimensional data by proposing a memory-bounded, hybrid CPU-GPU architecture. It introduces a multi-stage pipeline where a GPU piloting stage operates on SVD-reduced vectors to rapidly explore a subgraph, followed by CPU-based refinement with full vectors and a final complete traversal, aided by a fast entry selection mechanism. The approach yields substantial throughput gains (3.9–5.4×) on 100-million-scale datasets and scales beyond GPU memory by up to 12× using a single commodity GPU, with an open-source release. This design preserves search quality, demonstrates practical impact for retrieval tasks in AI systems, and offers a scalable path for deploying high-performance ANNS on accessible hardware.

Abstract

Approximate Nearest Neighbor Search (ANNS) has become fundamental to modern deep learning applications, having gained particular prominence through its integration into recent generative models that work with increasingly complex datasets and higher vector dimensions. Existing CPU-only solutions, even the most efficient graph-based ones, struggle to meet these growing computational demands, while GPU-only solutions face memory constraints. As a solution, we propose PilotANN, a hybrid CPU-GPU system for graph-based ANNS that utilizes both CPU's abundant RAM and GPU's parallel processing capabilities. Our approach decomposes the graph traversal process of top-$k$ search into three stages: GPU-accelerated subgraph traversal using SVD-reduced vectors, CPU refinement and precise search using complete vectors. Furthermore, we introduce fast entry selection to improve search starting points while maximizing GPU utilization. Experimental results demonstrate that PilotANN achieves $3.9 - 5.4 \times$ speedup in throughput on 100-million scale datasets, and is able to handle datasets up to $12 \times$ larger than the GPU memory. We offer a complete open-source implementation at https://github.com/ytgui/PilotANN.

Figures (11)

• Figure 1: Recall@10=0.90 QPS on 100 million datasets.
• Figure 2: Architecture of PilotANN.
• Figure 3: Calculation requirements of LAION-1M under different $\frac{\tau}{ef}$ conditions.
• Figure 4: Acceleration thresholds to obtain $2 \times$ distance calculation savings.
• Figure 5: Recall-Throughput curves on 100 million datasets.