HAKES: Scalable Vector Database for Embedding Search Service

TL;DR

HAKES tackles the challenge of scalable, high-recall embedding search under concurrent read-write workloads. It introduces a two-stage filter-refine index with learned compression parameters and an early-termination mechanism, plus a disaggregated distributed architecture that separates filter indexing from full-vector refinement. The approach combines dimensionality reduction, IVF-PQ partitioning, and self-supervised training to optimize the similarity score distribution locally, enabling updates without reindexing and maintaining high throughput. Empirical results show up to 16x throughput gains over baselines at high recall across diverse high-dimensional datasets, with strong scalability as nodes increase. This work offers a practical path to production-grade vector databases for retrieval-augmented generation and related embedding-search tasks.

Abstract

Modern deep learning models capture the semantics of complex data by transforming them into high-dimensional embedding vectors. Emerging applications, such as retrieval-augmented generation, use approximate nearest neighbor (ANN) search in the embedding vector space to find similar data. Existing vector databases provide indexes for efficient ANN searches, with graph-based indexes being the most popular due to their low latency and high recall in real-world high-dimensional datasets. However, these indexes are costly to build, suffer from significant contention under concurrent read-write workloads, and scale poorly to multiple servers. Our goal is to build a vector database that achieves high throughput and high recall under concurrent read-write workloads. To this end, we first propose an ANN index with an explicit two-stage design combining a fast filter stage with highly compressed vectors and a refine stage to ensure recall, and we devise a novel lightweight machine learning technique to fine-tune the index parameters. We introduce an early termination check to dynamically adapt the search process for each query. Next, we add support for writes while maintaining search performance by decoupling the management of the learned parameters. Finally, we design HAKES, a distributed vector database that serves the new index in a disaggregated architecture. We evaluate our index and system against 12 state-of-the-art indexes and three distributed vector databases, using high-dimensional embedding datasets generated by deep learning models. The experimental results show that our index outperforms index baselines in the high recall region and under concurrent read-write workloads. Furthermore, \namesys{} is scalable and achieves up to $16\times$ higher throughputs than the baselines. The HAKES project is open-sourced at https://www.comp.nus.edu.sg/~dbsystem/hakes/.

Figures (26)

• Figure 1: Vector database in retrieval augmented generation.
• Figure 2: Graph-based ANN index.
• Figure 3: Partitioning-based ANN index.
• Figure 4: HAKES-Index overview.
• Figure 5: End-to-end index construction.