Harnessing the Universal Geometry of Embeddings

TL;DR

The paper tackles translating text embeddings across heterogeneous spaces without paired data by introducing vec2vec, which learns a universal latent representation via adapters, a shared backbone, and cycle-consistent adversarial training with reconstruction and vector-space preservation. It demonstrates near-perfect alignment across model pairs and robustness to out-of-distribution inputs, enabling information extraction and inversion from translated embeddings. The work also uncovers potential security risks: an adversary could extract sensitive information from embedding databases by translating to a known space. It provides empirical support for a Strong Platonic Representation Hypothesis in text and suggests extensions to multi-modal embeddings like CLIP.

Abstract

We introduce the first method for translating text embeddings from one vector space to another without any paired data, encoders, or predefined sets of matches. Our unsupervised approach translates any embedding to and from a universal latent representation (i.e., a universal semantic structure conjectured by the Platonic Representation Hypothesis). Our translations achieve high cosine similarity across model pairs with different architectures, parameter counts, and training datasets. The ability to translate unknown embeddings into a different space while preserving their geometry has serious implications for the security of vector databases. An adversary with access only to embedding vectors can extract sensitive information about the underlying documents, sufficient for classification and attribute inference.

Figures (7)

• Figure 1: Left: input embeddings from different model families (T5-based GTR ni2021gtr and BERT-based GTE li2023gte) are fundamentally incomparable. Right: given unpaired embedding samples from different models on different texts, our model learns a latent representation where they are closely aligned.
• Figure 2: Given only a vector database from an unknown model, vec2vec translates the database into the space of a known model using latent structure alone. Converted embeddings reveal sensitive information about the original documents, such as the topic of an email (pictured, real example).
• Figure 3: Unsupervised embedding translation. With access to only $u_i = M_1(d_i)$, vec2vec seeks to generate a translation $F(u_i)$ that is close in $M_2$'s embedding space to the ideal embedding $v_i = M_2(d_i)$ without access to $d_i$, $v_i$, or $M_1$.
• Figure 4: Pairwise cosine similarities of input embeddings (left) and their vec2vec latents (middle) across different embedding pairs. The absolute difference between the heatmaps plots is on the right. All numbers are computed on the same batch of 1024 NQ texts.
• Figure 5: Leakage of information via inversion. Trained on NQ and evaluated on a 50-email subset of the Enron Email Corpus. Cells denote judge accuracy.