QPAD: Quantile-Preserving Approximate Dimension Reduction for Nearest Neighbors Preservation in High-Dimensional Vector Search

TL;DR

QPAD addresses the problem of deploying dimensionality reduction in high-dimensional vector search without sacrificing nearest-neighbor fidelity. It introduces a margin-based, unsupervised objective that preserves true $k$-NN relationships by maximizing the lower-tail distance gaps while employing a soft orthogonality penalty to diversify projection directions. The paper provides both Naive-QPAD and Fast-QPAD algorithms, with Fast-QPAD achieving $O(N\log N + Nn)$ per-evaluation complexity and strong theoretical guarantees, including Lipschitz continuity and KKT stationarity, as well as quantization-stable neighborhood preservation. Empirically, QPAD consistently outperforms 11 standard DR methods across five real-world domains and remains highly indexable across several ANN backends, demonstrating practical impact for scalable, retrieval-centric vector databases.

Abstract

High-dimensional vector embeddings are widely used in retrieval systems, but they often suffer from noise, the curse of dimensionality, and slow runtime. However, dimensionality reduction (DR) is rarely applied due to its tendency to distort the nearest-neighbor (NN) structure that is critical for search. Existing DR techniques such as PCA and UMAP optimize global or manifold-preserving criteria, rather than retrieval-specific objectives. We present QPAD -- Quantile-Preserving Approximate Dimension Reduction, an unsupervised DR method that explicitly preserves approximate NN relations by maximizing the margin between k-NNs and non-k-NNs under a soft orthogonality constraint. We analyze its complexity and favorable properties. This design enables QPAD to retain ANN-relevant geometry without supervision or changes to the original embedding model, while supporting scalability for large-scale vector search and being indexable for ANN search. Experiments across five domains show that QPAD consistently outperforms eleven standard DR methods in preserving neighborhood structure, enabling more accurate search in reduced dimensions.

Figures (5)

• Figure 1: Average $\overline{Recall@k}$ across all Dimensionality Retention Ratios and neighborhood sizes for each dataset under a fixed parameter combination, $\alpha=1$, $b=70$. QPAD consistently achieves the highest or second-highest performance compared to baseline methods.
• Figure 2: Ablation Study. Each subplot visualizes the effect of varying one parameter (e.g., $k$, Dimensionality Retention Ratio, $b$, or $\alpha$), with all others fixed. QPAD consistently maintains strong performance across a broad parameter range. The baseline parameter combinations are listed for each dataset.
• Figure 3: $Recall@10$ vs. dataset size under different indexing and quantization backends on FastText. Reduced from 300 dimensions to 128 dimensions. The effect of Dimension Retention Ratio on $Recall@k$ can be found in the ablation study, section \ref{['subsec:ablation']}. We vary the train set size from 10k to 100k vectors; the test set contains approximately 25% of the train size. k-NN means that exact k-NN result (brute-force, without indexing).
• Figure 4: Dimensionality reduction runtime scaling (normalized to the runtime at 10k). For QPAD, the absolute runtimes are 165.7s (10k), 518.1s (50k), and 875.6s (100k). k-NN means that exact k-NN result (brute-force, without indexing).
• Figure 5: $Recall@k$ vs. $k$ on SIFT-1M. Reduced from 128 dimensions to 64 dimensions. We only included the worst-case IVF $Recall@k$ result (IVF-OPQ-PQ) selected from figure \ref{['fig:scalability_recall']} to represent this indexing method.

Theorems & Definitions (5)

• proof : Proof of Boundedness.
• proof : Proof of Compactness.
• proof : Proof of Continuity.
• proof : Proof of Monotonicity.
• proof : Proof of Signed Measure Properties.