Characterizing the Dilemma of Performance and Index Size in Billion-Scale Vector Search and Breaking It with Second-Tier Memory

TL;DR

The paper tackles the bottleneck where high-performance vector search on billion-scale data clashes with the large index sizes required on SSDs. It argues that second-tier memory with fine-grained, robust IO patterns can address this mismatch. The authors design and evaluate improved graph and cluster indexes tailored to second-tier memory, achieving orders-of-magnitude reductions in index amplification and substantial throughput gains across RDMA, CXL, and NVM devices. Their end-to-end comparisons reveal when to favor graph versus cluster indexes on second-tier memory, offering practical guidance for developers deploying vector search systems at scale.

Abstract

Vector searches on large-scale datasets are critical to modern online services like web search and RAG, which necessity storing the datasets and their index on the secondary storage like SSD. In this paper, we are the first to characterize the trade-off of performance and index size in existing SSD-based graph and cluster indexes: to improve throughput by 5.7$\times$ and 1.7$\times$, these indexes have to pay a 5.8$\times$ storage amplification and 7.7$\times$ with respect to the dataset size, respectively. The root cause is that the coarse-grained access of SSD mismatches the fine-grained random read required by vector indexes with small amplification. This paper argues that second-tier memory, such as remote DRAM/NVM connected via RDMA or CXL, is a powerful storage for addressing the problem from a system's perspective, thanks to its fine-grained access granularity. However, putting existing indexes -- primarily designed for SSD -- directly on second-tier memory cannot fully utilize its power. Meanwhile, second-tier memory still behaves more like storage, so using it as DRAM is also inefficient. To this end, we build a graph and cluster index that centers around the performance features of second-tier memory. With careful execution engine and index layout designs, we show that vector indexes can achieve optimal performance with orders of magnitude smaller index amplification, on a variety of second-tier memory devices. Based on our improved graph and vector indexes on second-tier memory, we further conduct a systematic study between them to facilitate developers choosing the right index for their workloads. Interestingly, the findings on the second-tier memory contradict the ones on SSDs.

Figures (13)

• Figure 1: An illustration of (a) the basic data structures of graph-based vector index, (b) the execution flow of vector search with it and (c) how the data structures are stored.
• Figure 2: An illustration of (a) the basic data structures of cluster-based vector index, (b) the execution flow of vector search with it and (c) how the data structures are stored.
• Figure 3: First-tier memory vs. second-tier memory.
• Figure 4: (a) Performance—index size trade-off in existing graph-based (upper) and cluster-based (bottom) vector indexes. The amplification is normalized to the total dataset size without indexing. (b) I/O dominates index performance across various datasets. (c) The performance increases when the index size becomes larger is due to the reduced number of I/O sent per-query and (d) the request payload per I/O and its trend compared to the reduced number of I/O per query. Note that all the performance data is normalized.
• Figure 5: An illustration of (a) the index size amplification problem in graph indexing and (b) why a cluster index requires replication.