Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Elastic Index Selection for Label-Hybrid AKNN Search

Mingyu Yang, Wenxuan Xia, Wentao Li, Raymond Chi-Wing Wong, Wei Wang

TL;DR

<3-5 sentence high-level summary> Addresses the challenge of scalable label-hybrid AKNN by showing that indexes built for a query-label set can serve supersets via label containment, bounded by an elastic factor. Introduces the Elastic Index Selection problem, proves NP-completeness, and provides greedy algorithms for both efficiency- and space-constrained variants. Demonstrates through extensive experiments that selective indexing achieves near-optimal search performance with strong space-efficiency trade-offs and substantial speedups over state-of-the-art baselines. The approach integrates with existing graph-based AKNN libraries and supports dynamic updates, offering a practical path to scalable, workload-aware label-hybrid ANN systems.

Abstract

Real-world vector embeddings are usually associated with extra labels, such as attributes and keywords. Many applications require the nearest neighbor search that contains specific labels, such as searching for product image embeddings restricted to a particular brand. A straightforward approach is to materialize all possible indices according to the complete query label workload. However, this leads to an exponential increase in both index space and processing time, which significantly limits scalability and efficiency. In this paper, we leverage the inclusion relationships among query label sets to construct partial indexes, enabling index sharing across queries for improved construction efficiency. We introduce \textit{elastic factor} bounds to guarantee search performance and use the greedy algorithm to select indices that meet the bounds, achieving a tradeoff between efficiency and space. Meanwhile, we also designed the algorithm to achieve the best elastic factor under a given space limitation. Experimental results on multiple real datasets demonstrate that our algorithm can achieve near-optimal search performance, achieving up to 10x-500x search efficiency speed up over state-of-the-art approaches. Our algorithm is highly versatile, since it is not constrained by index type and can seamlessly integrate with existing optimized libraries.

Elastic Index Selection for Label-Hybrid AKNN Search

TL;DR

<3-5 sentence high-level summary> Addresses the challenge of scalable label-hybrid AKNN by showing that indexes built for a query-label set can serve supersets via label containment, bounded by an elastic factor. Introduces the Elastic Index Selection problem, proves NP-completeness, and provides greedy algorithms for both efficiency- and space-constrained variants. Demonstrates through extensive experiments that selective indexing achieves near-optimal search performance with strong space-efficiency trade-offs and substantial speedups over state-of-the-art baselines. The approach integrates with existing graph-based AKNN libraries and supports dynamic updates, offering a practical path to scalable, workload-aware label-hybrid ANN systems.

Abstract

Real-world vector embeddings are usually associated with extra labels, such as attributes and keywords. Many applications require the nearest neighbor search that contains specific labels, such as searching for product image embeddings restricted to a particular brand. A straightforward approach is to materialize all possible indices according to the complete query label workload. However, this leads to an exponential increase in both index space and processing time, which significantly limits scalability and efficiency. In this paper, we leverage the inclusion relationships among query label sets to construct partial indexes, enabling index sharing across queries for improved construction efficiency. We introduce \textit{elastic factor} bounds to guarantee search performance and use the greedy algorithm to select indices that meet the bounds, achieving a tradeoff between efficiency and space. Meanwhile, we also designed the algorithm to achieve the best elastic factor under a given space limitation. Experimental results on multiple real datasets demonstrate that our algorithm can achieve near-optimal search performance, achieving up to 10x-500x search efficiency speed up over state-of-the-art approaches. Our algorithm is highly versatile, since it is not constrained by index type and can seamlessly integrate with existing optimized libraries.
Paper Structure (17 sections, 2 theorems, 1 equation, 21 figures, 6 tables, 1 algorithm)

This paper contains 17 sections, 2 theorems, 1 equation, 21 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminary
  3. Problem Statement
  4. Existing Solutions
  5. Elastic Index Selection Problem
  6. Elastic Factor for Index-Sharing
  7. Problem Definition
  8. Problem Hardness
  9. Problem Solution
  10. Method for Efficiency-constrained Variant
  11. Method for Space-Constrained Variant
  12. Discussions
  13. Experiment
  14. Experiment Settings
  15. Experiment Results
  16. ...and 2 more sections

Key Result

lemma 1

Given a dataset $S$, a query $(q, L_q)$, and a selected index set $\mathbb{I}$, let $O(C)$ denote the expected time to retrieve the top-1 neighbor from a graph index. If the elastic factor satisfies $e(S(L_q), \mathbb{I}) \ge c$ for some constant $c \in (0,1]$, then the expected time to obtain the t

Figures (21)

  • Figure 1: Example of label-hybrid AKNN search in an online shopping scenario ($k=1$). Each item is represented by an embedding vector $x_i$ ($i \in [1,4]$) and associated with label attributes such as brand and year. A customer submits a reference photo $q$ together with a query-label set $L_q$. The task is to find the item most similar to $q$ that also satisfies the label constraint. Thus, $x_2$ and $x_3$ are discarded for label mismatch, and $x_4$ is returned as the nearest neighbor.
  • Figure 2: Motivating example where database $S$ and queries draw labels from the alphabet $\{A,B,C\}$. Base vectors in $S$ are grouped by their associated label sets $L$, each group denoted as $\{L\}$[count]. Arrows connect a group $L$ to groups with its minimal supersets. For instance, Fig. \ref{['fig:motivate-exmaple']}(a) shows $400$ vectors labeled only $\{A\}$, written as $A[400]$. Given a AKNN query with $L_q=\{A\}$, Fig. \ref{['fig:motivate-exmaple']}(b) highlights all relevant groups---those whose label sets are supersets of $\{A\}$---namely $\{A,AB,AC,ABC\}$, totaling $1,000$ matching vectors that must be indexed under $L_q=\{A\}$. Fig. \ref{['fig:motivate-exmaple']}(c) summarizes the index sizes required for each query-label set. Supporting all possible query-label sets in the workload requires indexing $5,400$ vectors---about $2.75\times$ the original dataset size.
  • Figure 3: The Example of Filter-Based Search
  • Figure 4: The Running Example of Elastic Factor and Greedy Methods
  • Figure 5: Effect of the elastic factor on query efficiency ($k=10$: left, $k=20$: right). We randomly generate label sets for both base and query vectors and build ${ {\mathsf{HNSW}}} \xspace$ indexes on the SIFT100M dataset. Queries are grouped by elastic factor---0.001, 0.01, 0.1, and 1---with $e=1$ serving as the optimal baseline.
  • ...and 16 more figures

Theorems & Definitions (10)

  • definition 1: Label-Hybrid KNN Search
  • definition 2: Elastic Factor
  • Example 1
  • lemma 1
  • definition 3: Fixed Efficiency Index Selection (${ {\mathsf{EIS}}} \xspace$)
  • Example 2
  • theorem 1
  • definition 4
  • Example 3
  • definition 5: 3-Set Cover (3-${ {\mathsf{SC}}} \xspace$)