Spectral Tempering for Embedding Compression in Dense Passage Retrieval

Abstract

Dimensionality reduction is critical for deploying dense retrieval systems at scale, yet mainstream post-hoc methods face a fundamental trade-off: principal component analysis (PCA) preserves dominant variance but underutilizes representational capacity, while whitening enforces isotropy at the cost of amplifying noise in the heavy-tailed eigenspectrum of retrieval embeddings. Intermediate spectral scaling methods unify these extremes by reweighting dimensions with a power coefficient $γ$, but treat $γ$ as a fixed hyperparameter that requires task-specific tuning. We show that the optimal scaling strength $γ$ is not a global constant: it varies systematically with target dimensionality $k$ and is governed by the signal-to-noise ratio (SNR) of the retained subspace. Based on this insight, we propose Spectral Tempering (\textbf{SpecTemp}), a learning-free method that derives an adaptive $γ(k)$ directly from the corpus eigenspectrum using local SNR analysis and knee-point normalization, requiring no labeled data or validation-based search. Extensive experiments demonstrate that Spectral Tempering consistently achieves near-oracle performance relative to grid-searched $γ^*(k)$ while remaining fully learning-free and model-agnostic. Our code is publicly available at https://anonymous.4open.science/r/SpecTemp-0D37.

Figures (2)

• Figure 1: Consistent spectral structure of dense retrieval embeddings. Eigenvalue distributions from 1M sampled embeddings on MS MARCO and NQ exhibit consistent heavy-tailed decay across diverse retrievers, revealing a head–tail signal-to-noise ratio (SNR) gradient—leading components are signal-dominant while tail dimensions grow noise-prone—motivating dimensionality-adaptive tempering.
• Figure 2: Performance consistency across additional models.