Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Lossless Compression of Vector IDs for Approximate Nearest Neighbor Search

Daniel Severo, Giuseppe Ottaviano, Matthew Muckley, Karen Ullrich, Matthijs Douze

TL;DR

The paper tackles the RAM bottleneck in approximate nearest neighbor search by focusing on lossless compression of vector IDs and graph edges in IVF and graph indices. It presents ANS-based codecs and wavelet-tree techniques that exploit order invariances to represent IDs and edges as sets, achieving large bitrate reductions with negligible impact on search speed. Key contributions include practical lossless codecs for IDs (ROC and WT/RS options) and a graph-oriented REC approach, plus an offline option for entire databases. The approach yields substantial savings on large-scale datasets (e.g., ~7x for IVF, up to 50% for graphs) and even enables conditioning-based compression of quantized vector codes, with open-source code available for integration into FAISS pipelines. Overall, the work significantly reduces index footprint while preserving exact search behavior, enabling scalable ANNS on bigger datasets."

Abstract

Approximate nearest neighbor search for vectors relies on indexes that are most often accessed from RAM. Therefore, storage is the factor limiting the size of the database that can be served from a machine. Lossy vector compression, i.e., embedding quantization, has been applied extensively to reduce the size of indexes. However, for inverted file and graph-based indices, auxiliary data such as vector ids and links (edges) can represent most of the storage cost. We introduce and evaluate lossless compression schemes for these cases. These approaches are based on asymmetric numeral systems or wavelet trees that exploit the fact that the ordering of ids is irrelevant within the data structures. In some settings, we are able to compress the vector ids by a factor 7, with no impact on accuracy or search runtime. On billion-scale datasets, this results in a reduction of 30% of the index size. Furthermore, we show that for some datasets, these methods can also compress the quantized vector codes losslessly, by exploiting sub-optimalities in the original quantization algorithm. The source code for our approach available at https://github.com/facebookresearch/vector_db_id_compression.

Lossless Compression of Vector IDs for Approximate Nearest Neighbor Search

TL;DR

The paper tackles the RAM bottleneck in approximate nearest neighbor search by focusing on lossless compression of vector IDs and graph edges in IVF and graph indices. It presents ANS-based codecs and wavelet-tree techniques that exploit order invariances to represent IDs and edges as sets, achieving large bitrate reductions with negligible impact on search speed. Key contributions include practical lossless codecs for IDs (ROC and WT/RS options) and a graph-oriented REC approach, plus an offline option for entire databases. The approach yields substantial savings on large-scale datasets (e.g., ~7x for IVF, up to 50% for graphs) and even enables conditioning-based compression of quantized vector codes, with open-source code available for integration into FAISS pipelines. Overall, the work significantly reduces index footprint while preserving exact search behavior, enabling scalable ANNS on bigger datasets."

Abstract

Approximate nearest neighbor search for vectors relies on indexes that are most often accessed from RAM. Therefore, storage is the factor limiting the size of the database that can be served from a machine. Lossy vector compression, i.e., embedding quantization, has been applied extensively to reduce the size of indexes. However, for inverted file and graph-based indices, auxiliary data such as vector ids and links (edges) can represent most of the storage cost. We introduce and evaluate lossless compression schemes for these cases. These approaches are based on asymmetric numeral systems or wavelet trees that exploit the fact that the ordering of ids is irrelevant within the data structures. In some settings, we are able to compress the vector ids by a factor 7, with no impact on accuracy or search runtime. On billion-scale datasets, this results in a reduction of 30% of the index size. Furthermore, we show that for some datasets, these methods can also compress the quantized vector codes losslessly, by exploiting sub-optimalities in the original quantization algorithm. The source code for our approach available at https://github.com/facebookresearch/vector_db_id_compression.
Paper Structure (42 sections, 5 equations, 3 figures, 4 tables)

This paper contains 42 sections, 5 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Orderless compression.
  3. Related Work
  4. Vector compression.
  5. Vector search
  6. Background
  7. Notation
  8. Lossless Compression
  9. Asymmetric Numeral Systems (ANS)
  10. Bits-back for Combinatorial Objects
  11. Bits-back Coding
  12. Initial bits issue.
  13. Random Order Coding (ROC)
  14. Random Edge Coding (REC)
  15. Wavelet tree codecs
  16. ...and 27 more sections

Figures (3)

  • Figure 1: How we apply lossless compression in state-of-the-art indexes for vector search. (Orange) Embeddings/vectors are compressed using lossy techniques such as Product Quantization jegou2010product and QINCo niu2023residualhuijben2024QINco. (Green) We focus on lossless compression of identifiers (IVF, top) and links (edges in graph indices, bottom).
  • Figure 2: Slowdown relative to the Uncompressed index. As the dimensionality of PQ increases, the overall slowdown is less significant.
  • Figure 3: Results for compressing quantization codes conditioned on clusters (originally $8$ bits-per-element). Lower is better.