More Than Efficiency: Embedding Compression Improves Domain Adaptation in Dense Retrieval

TL;DR

The paper tackles domain shift in dense retrieval by applying unsupervised PCA-based embedding compression to target-domain queries (and optionally documents). By projecting high-dimensional embeddings into a lower-dimensional subspace defined by the top principal components, the method reorients retrieval toward domain-relevant information without any labeled data or fine-tuning. Across 9 pretrained models and 14 MTEB datasets, query-only PCA improves $NDCG@10$ in 75.4% of model-dataset pairs, often with retention ratios between 50% and 90% of the original dimensions, and shows competitive performance relative to heavier domain-adaptation pipelines. The work demonstrates a simple, zero-cost baseline that can serve as a first-step domain adaptation technique, with implications for scalable retrieval in specialized domains and potential extensions to nonlinear manifold methods.

Abstract

Dense retrievers powered by pretrained embeddings are widely used for document retrieval but struggle in specialized domains due to the mismatches between the training and target domain distributions. Domain adaptation typically requires costly annotation and retraining of query-document pairs. In this work, we revisit an overlooked alternative: applying PCA to domain embeddings to derive lower-dimensional representations that preserve domain-relevant features while discarding non-discriminative components. Though traditionally used for efficiency, we demonstrate that this simple embedding compression can effectively improve retrieval performance. Evaluated across 9 retrievers and 14 MTEB datasets, PCA applied solely to query embeddings improves NDCG@10 in 75.4% of model-dataset pairs, offering a simple and lightweight method for domain adaptation.

Figures (5)

• Figure 1: PCA compresses embeddings to become domain-specific, providing a more efficient domain adaptation method than traditional fine-tuning, with an additional advantage of a lower retrieval cost.
• Figure 2: Pipeline for adapting embeddings to a test time domain using PCA (§\ref{['subsec:pca']}). The encoder in the top branch processes text samples (queries, documents, or a mixture) from the test domain. These samples are used by the PCA algorithm to learn a projection matrix of the top $d'$ principal components. This matrix then transforms both the test domain's query and document embeddings. Finally, a similarity match is performed on the transformed embeddings to retrieve the most relevant documents.
• Figure 3: The distribution of the eigenvalues for the principal components after applying PCA to fit the queries' embeddings of the Sentence-T5 model on each dataset (ArguAna and SciDocs).
• Figure 4: NDCG@10 at various retention ratios ranging from 0.1 to 1, with step size of 0.1 and an additional inclusion of a very extreme 0.05. While the optimal Query retention ratio is dataset- and model-dependent, in most cases where Query Compression dominates the original embeddings, the dominance can continue even until the retention ratio is around 50%.
• Figure 5: Correlation between domain familiarity (Equation \ref{['eq:domainfamiliarity']}) and performance improvement after 90% Query Compression. For BGE, Instructor, and Dis-Ro, retrieval performance after compression benefits from higher domain familiarity.