Curator: Efficient Indexing for Multi-Tenant Vector Databases

TL;DR

Curator is presented, an in-memory vector index design tailored for multi-tenant queries that simultaneously achieves the two conflicting goals, low memory overhead and high performance for queries, vector insertion, and deletion.

Abstract

Vector databases have emerged as key enablers for bridging intelligent applications with unstructured data, providing generic search and management support for embedding vectors extracted from the raw unstructured data. As multiple data users can share the same database infrastructure, multi-tenancy support for vector databases is increasingly desirable. This hinges on an efficient filtered search operation, i.e., only querying the vectors accessible to a particular tenant. Multi-tenancy in vector databases is currently achieved by building either a single, shared index among all tenants, or a per-tenant index. The former optimizes for memory efficiency at the expense of search performance, while the latter does the opposite. Instead, this paper presents Curator, an in-memory vector index design tailored for multi-tenant queries that simultaneously achieves the two conflicting goals, low memory overhead and high performance for queries, vector insertion, and deletion. Curator indexes each tenant's vectors with a tenant-specific clustering tree and encodes these trees compactly as sub-trees of a shared clustering tree. Each tenant's clustering tree adapts dynamically to its unique vector distribution, while maintaining a low per-tenant memory footprint. Our evaluation, based on two widely used data sets, confirms that Curator delivers search performance on par with per-tenant indexing, while maintaining memory consumption at the same level as metadata filtering on a single, shared index.

Figures (15)

• Figure 1: Current multi-tenancy strategies incur inherent inefficiency in either query latency or memory usage, while Curator aims to optimize both simultaneously.
• Figure 2: Statistics of the YFCC100M dataset. (a) Most tenants have access to only a small portion of all vectors. (b) On average, each vector is accessible to around 10 tenants, although some vectors are shared among up to 100 tenants.
• Figure 3: Shared index fails to capture the unique vector distribution of individual tenants, compromising search efficiency. Vectors of the querying tenant are highlighted in red, while others are in gray.
• Figure 4: Overview of Curator. For simplicity, we represent the high-dimensional vector space as a line and restrict the branching factor of each tree node to 2. For a tree node $n$ in the Tenant Clustering Tree of tenant $t$, BF and SL in the legend stand for whether the Bloom filter of $n$ contains $t$ and whether $n$ contains a shortlist for $t$, respectively.
• Figure 5: $k$-ANN similarity search on Curator.