When Embedding Models Meet: Procrustes Bounds and Applications

TL;DR

The paper addresses interoperability gaps between independently trained embedding models by showing that approximate preservation of pairwise dot products implies the existence of a close orthogonal Procrustes alignment between spaces, with a tight bound $\min_{\bm{Q} \in \mathcal{O}_D} \lVert \bm{Q}\bm{X} - \bm{Y} \rVert_F \le (2D)^{1/4} \sqrt{\varepsilon}$ where $\varepsilon = \lVert \bm{X}^\top\bm{X} - \bm{Y}^\top\bm{Y} \rVert_F$. This yields a simple Procrustes post-processing recipe to make embeddings interoperable without distorting each space's geometry. The authors validate the approach across three practical applications—maintaining compatibility across retrainings, combining different text models for retrieval, and improving mixed-modality search—demonstrating improved performance and, in some cases, state-of-the-art results. Overall, the work provides a solid theoretical guarantee for embedding interoperability and a practical tool for real-world multimodel pipelines that rely on cross-model similarity computations.

Abstract

Embedding models trained separately on similar data often produce representations that encode stable information but are not directly interchangeable. This lack of interoperability raises challenges in several practical applications, such as model retraining, partial model upgrades, and multimodal search. Driven by these challenges, we study when two sets of embeddings can be aligned by an orthogonal transformation. We show that if pairwise dot products are approximately preserved, then there exists an isometry that closely aligns the two sets, and we provide a tight bound on the alignment error. This insight yields a simple alignment recipe, Procrustes post-processing, that makes two embedding models interoperable while preserving the geometry of each embedding space. Empirically, we demonstrate its effectiveness in three applications: maintaining compatibility across retrainings, combining different models for text retrieval, and improving mixed-modality search, where it achieves state-of-the-art performance.

Figures (10)

• Figure 1: We start with two sets of embeddings $\bm{X}$ and $\bm{Y}$ that approximately preserve distances but are unaligned (left). We find $\bar{\bm{X}}$ by apply the orthogonal Procrustes transformation to $\bm{X}$ (center). $\bar{\bm{X}}$ retains the exact geometry of $\bm{X}$ but the embeddings are now aligned with $\bm{Y}$ (right).
• Figure 2: Retraining experiments on the MovieLens dataset. Models trained on embeddings from version $t_0$ are combined with embeddings from version $t > t_0$.
• Figure 3: Two-dimensional representation of the text embedding models reflecting approximate Procrustes distances (left). Normalized Procrustes distance vs. geometry-preservation for all 21.0 pairwise model combinations (right).
• Figure 4: Retrieval performance (nDCG@10) for all query--document model combinations. Top: raw embeddings. Bottom: query embeddings aligned with orthogonal Procrustes. Diagonal entries correspond to the baseline case where the same model is used for both queries and documents.
• Figure 5: Difference in nDCG@10 between orthogonal Procrustes and unconstrained linear alignment. Positive values indicate orthogonal Procrustes performs better.

Theorems & Definitions (10)

• Theorem 1
• proof : Proof (sketch).
• Corollary 1
• proof
• Corollary 2
• proof
• Lemma 1: Powers-Størmers-Kittaneh Inequality
• Lemma 2
• proof
• proof : Proof of Theorem \ref{['thm:main']}