Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RAIRS: Optimizing Redundant Assignment and List Layout for IVF-Based ANN Search

Zehai Yang, Shimin Chen

TL;DR

RAIRS tackles redundant assignment in IVF based ANNS by introducing AIR for optimized list selection and SEIL for shared cell layout to ease duplicate distance computations. AIR uses the loss L(c', c, Q) ∝ ||r'||^2 + $\lambda$ r^T r' with $r=c-x$ and $r'=c'-x$ to encourage selecting a second list whose residual is opposite to the first, improving recall. SEIL shares blocks of large cells across lists to avoid duplicating distance computations and employs cell level dedup with a listVisited mechanism, while RAIR selects secondary lists using the AIR metric. Empirical results across multiple real world datasets show RAIRS achieving up to 1.33x throughput gains over IVFPQfs with refinement and maintaining recall, demonstrating broad applicability to Euclidean space IVF based ANN search and beyond.

Abstract

IVF is one of the most widely used ANNS (Approximate Nearest Neighbors Search) methods in vector databases. The idea of redundant assignment is to assign a data vector to more than one IVF lists for reducing the chance of missing true neighbors in IVF search. However, the naive strategy, which selects the second IVF list based on the distance between a data vector and the list centroids, performs poorly. Previous work focuses only on the inner product distance, while there is no optimized list selection study for the most popular Euclidean space. Moreover, the IVF search may access the same vector in more than one lists, resulting in redundant distance computation and decreasing query throughput. In this paper, we present RAIRS to address the above two challenges. For the challenge of the list selection, we propose an optimized AIR metric for the Euclidean space. AIR takes not only distances but also directions into consideration in order to support queries that are closer to the data vector but father away from the first chosen list's centroid. For the challenge of redundant distance computation, we propose SEIL, an optimized list layout that exploits shared cells to reduce repeated distance computations for IVF search. Our experimental results using representative real-world data sets show that RAIRS out-performs existing redundant assignment solutions and achieves up to 1.33x improvement over the best-performing IVF method, IVF-PQ Fast Scan with refinement.

RAIRS: Optimizing Redundant Assignment and List Layout for IVF-Based ANN Search

TL;DR

RAIRS tackles redundant assignment in IVF based ANNS by introducing AIR for optimized list selection and SEIL for shared cell layout to ease duplicate distance computations. AIR uses the loss L(c', c, Q) ∝ ||r'||^2 + r^T r' with and to encourage selecting a second list whose residual is opposite to the first, improving recall. SEIL shares blocks of large cells across lists to avoid duplicating distance computations and employs cell level dedup with a listVisited mechanism, while RAIR selects secondary lists using the AIR metric. Empirical results across multiple real world datasets show RAIRS achieving up to 1.33x throughput gains over IVFPQfs with refinement and maintaining recall, demonstrating broad applicability to Euclidean space IVF based ANN search and beyond.

Abstract

IVF is one of the most widely used ANNS (Approximate Nearest Neighbors Search) methods in vector databases. The idea of redundant assignment is to assign a data vector to more than one IVF lists for reducing the chance of missing true neighbors in IVF search. However, the naive strategy, which selects the second IVF list based on the distance between a data vector and the list centroids, performs poorly. Previous work focuses only on the inner product distance, while there is no optimized list selection study for the most popular Euclidean space. Moreover, the IVF search may access the same vector in more than one lists, resulting in redundant distance computation and decreasing query throughput. In this paper, we present RAIRS to address the above two challenges. For the challenge of the list selection, we propose an optimized AIR metric for the Euclidean space. AIR takes not only distances but also directions into consideration in order to support queries that are closer to the data vector but father away from the first chosen list's centroid. For the challenge of redundant distance computation, we propose SEIL, an optimized list layout that exploits shared cells to reduce repeated distance computations for IVF search. Our experimental results using representative real-world data sets show that RAIRS out-performs existing redundant assignment solutions and achieves up to 1.33x improvement over the best-performing IVF method, IVF-PQ Fast Scan with refinement.
Paper Structure (22 sections, 1 theorem, 4 equations, 18 figures, 4 tables, 5 algorithms)

This paper contains 22 sections, 1 theorem, 4 equations, 18 figures, 4 tables, 5 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. ANN Search
  4. IVF (Inverted File Index)
  5. Redundant Assignment
  6. RAIRS Overview
  7. Redundant List Selection with AIR
  8. Amplified Inverse Residual
  9. RAIR Algorithm
  10. Generalization to Multiple Assignments
  11. Shared-Cell Enhanced IVF Lists
  12. Challenges of Baseline List Layout
  13. SEIL Layout
  14. SEIL Algorithms
  15. Evaluation
  16. ...and 7 more sections

Key Result

theorem 1

For a set $Q$ of queries that are uniformly distributed in the hypersphere centered at $x$ with radius $l_m$, where $r=c-x$, $r'=c'-x$, and $\lambda>0$ is a constant factor.

Figures (18)

  • Figure 1: Redundant assignment for IVF index.
  • Figure 2: Different redundant assignment strategies.
  • Figure 3: Overview of the RAIRS index.
  • Figure 4: Geometric relationship of vectors.
  • Figure 5: Characteristics of cells after redundant assignment.
  • ...and 13 more figures

Theorems & Definitions (1)

  • theorem 1