Dimensionality-Reduction Techniques for Approximate Nearest Neighbor Search: A Survey and Evaluation

TL;DR

This paper tackles the challenge that ANNS distance calculations scale with the dimensionality $D$ of embeddings, particularly as modern models produce very high-dimensional vectors. It surveys six dimensionality-reduction techniques and two integration frameworks to accelerate ANNS, provides theoretical time/space analyses, and benchmarks their performance on six public datasets using a reproducible library, Fudist. The key finding is that carefully chosen DR methods can yield up to around $6\times$ speedups over baseline HNSW, though gains are dataset-dependent and trade-offs between accuracy and speed must be managed; out-of-place acceleration offers more flexibility for existing indexes, while in-place transformations can achieve lower overhead with re-indexing. The work provides practical guidance and opens directions for robust, provable acceleration through quantization, deep representations, adaptive technique selection, and storage-efficient vector representations.

Abstract

Approximate Nearest Neighbor Search (ANNS) on high-dimensional vectors has become a fundamental and essential component in various machine learning tasks. Recently, with the rapid development of deep learning models and the applications of Large Language Models (LLMs), the dimensionality of the vectors keeps growing in order to accommodate a richer semantic representation. This poses a major challenge to the ANNS solutions since distance calculation cost in ANNS grows linearly with the dimensionality of vectors. To overcome this challenge, dimensionality-reduction techniques can be leveraged to accelerate the distance calculation in the search process. In this paper, we investigate six dimensionality-reduction techniques that have the potential to improve ANNS solutions, including classical algorithms such as PCA and vector quantization, as well as algorithms based on deep learning approaches. We further describe two frameworks to apply these techniques in the ANNS workflow, and theoretically analyze the time and space costs, as well as the beneficial threshold for the pruning ratio of these techniques. The surveyed techniques are evaluated on six public datasets. The analysis of the results reveals the characteristics of the different families of techniques and provides insights into the promising future research directions.

Figures (4)

• Figure 1: Illustration of two frameworks to apply dimensionality-reduction techniques
• Figure 2: Query performance with dimensionality-reduction techniques.
• Figure 3: ANNS performance with SIMD (AVX)
• Figure 4: Approximation ratio on GIST (top) and H&M (bottom)