Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Universal Zero-shot Embedding Inversion

Collin Zhang, John X. Morris, Vitaly Shmatikov

TL;DR

This paper tackles embedding inversion by introducing ZSinvert, a universal zero-shot method that inverts text embeddings without training an embedding-specific model. It combines adversarial decoding with a staged refinement pipeline and an offline, encoder-agnostic correction model to improve lexical fidelity while preserving semantic similarity to the target embedding. Evaluations on MS-Marco and Enron demonstrate substantial semantic leakage from embeddings and reveal robustness to moderate Gaussian noise, highlighting security risks in vector databases and embedding-based retrieval. The approach is query-efficient and broadly applicable across current and future embedding models, with practical implications for data protection and privacy in retrieval systems.

Abstract

Embedding inversion, i.e., reconstructing text given its embedding and black-box access to the embedding encoder, is a fundamental problem in both NLP and security. From the NLP perspective, it helps determine how much semantic information about the input is retained in the embedding. From the security perspective, it measures how much information is leaked by vector databases and embedding-based retrieval systems. State-of-the-art methods for embedding inversion, such as vec2text, have high accuracy but require (a) training a separate model for each embedding, and (b) a large number of queries to the corresponding encoder. We design, implement, and evaluate ZSInvert, a zero-shot inversion method based on the recently proposed adversarial decoding technique. ZSInvert is fast, query-efficient, and can be used for any text embedding without training an embedding-specific inversion model. We measure the effectiveness of ZSInvert on several embeddings and demonstrate that it recovers key semantic information about the corresponding texts.

Universal Zero-shot Embedding Inversion

TL;DR

This paper tackles embedding inversion by introducing ZSinvert, a universal zero-shot method that inverts text embeddings without training an embedding-specific model. It combines adversarial decoding with a staged refinement pipeline and an offline, encoder-agnostic correction model to improve lexical fidelity while preserving semantic similarity to the target embedding. Evaluations on MS-Marco and Enron demonstrate substantial semantic leakage from embeddings and reveal robustness to moderate Gaussian noise, highlighting security risks in vector databases and embedding-based retrieval. The approach is query-efficient and broadly applicable across current and future embedding models, with practical implications for data protection and privacy in retrieval systems.

Abstract

Embedding inversion, i.e., reconstructing text given its embedding and black-box access to the embedding encoder, is a fundamental problem in both NLP and security. From the NLP perspective, it helps determine how much semantic information about the input is retained in the embedding. From the security perspective, it measures how much information is leaked by vector databases and embedding-based retrieval systems. State-of-the-art methods for embedding inversion, such as vec2text, have high accuracy but require (a) training a separate model for each embedding, and (b) a large number of queries to the corresponding encoder. We design, implement, and evaluate ZSInvert, a zero-shot inversion method based on the recently proposed adversarial decoding technique. ZSInvert is fast, query-efficient, and can be used for any text embedding without training an embedding-specific inversion model. We measure the effectiveness of ZSInvert on several embeddings and demonstrate that it recovers key semantic information about the corresponding texts.

Paper Structure

This paper contains 23 sections, 3 equations, 1 figure, 4 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Text embedding and RAG.
  4. Text inversion.
  5. Optimization-based language generation
  6. Threat Model
  7. Our Method: Embedding Inversion with Guided Generation, Progressive Refinement, and Correction
  8. Stage 1: Initial Seed Generation.
  9. Stage 2: Paraphrase-based Refinement.
  10. Stage 3: Correction using an Offline Model.
  11. Evaluation
  12. Encoders.
  13. Datasets.
  14. Correction Model.
  15. Evaluation metrics.
  16. ...and 8 more sections

Figures (1)

  • Figure 1: Text inversion performance increases as we increase number of correction iterations. The correction model is only trained on contriever candidate inversions, but it transfers to all other encoders.