Sentence Embedding Leaks More Information than You Expect: Generative Embedding Inversion Attack to Recover the Whole Sentence

TL;DR

The paper addresses privacy leakage from sentence embeddings produced by large language models by introducing GEIA, a generative, black-box inversion attack that reconstructs ordered sentences from embeddings using a decoder seeded with $Align(f(x))$. GEIA demonstrates superiority over prior embedding inversion methods by producing coherent sequences and recovering informative content, including named entities, across multiple embedding models and datasets. The work provides extensive experiments on PersonaChat and QNLI, showing improved classification and generation metrics and highlighting practical privacy risks. The findings motivate the development of defenses to protect sentence-level representations in real-world applications.

Abstract

Sentence-level representations are beneficial for various natural language processing tasks. It is commonly believed that vector representations can capture rich linguistic properties. Currently, large language models (LMs) achieve state-of-the-art performance on sentence embedding. However, some recent works suggest that vector representations from LMs can cause information leakage. In this work, we further investigate the information leakage issue and propose a generative embedding inversion attack (GEIA) that aims to reconstruct input sequences based only on their sentence embeddings. Given the black-box access to a language model, we treat sentence embeddings as initial tokens' representations and train or fine-tune a powerful decoder model to decode the whole sequences directly. We conduct extensive experiments to demonstrate that our generative inversion attack outperforms previous embedding inversion attacks in classification metrics and generates coherent and contextually similar sentences as the original inputs.

Figures (6)

• Figure 1: Overview of embedding inversion and attribute inference attacks on language models. Both attacks can be conducted on the sentence embedding $f(x)$. Previous embedding inversion attacks only predict sets of words while our generative embedding inversion attack is able to reconstruct actual input sequences.
• Figure 2: Model architecture for GEIA. The sentence embedding can be embedded from arbitrary pretrained sentence embedding models. The sentence embeddings are projected to the exact dimension of input token representations. After projection, the projected embeddings are concatenated with input representations to train the attacker. During inference, the sentence embeddings are fed as the initial token representations to decode corresponding inputs.
• Figure 3: Precision-recall curve of MLC on the PersonaChat dataset.
• Figure 4: Embedding inversion attacks' results on the victim embedding models on the PersonaChat dataset. We use "$\sqcup$" to denote the space and highlight some informative words. Given the same input sentence, the inverted results are shown. Both previous embedding inversion results can only invert unordered sets of predicted tokens while our generative embedding inversion can generate fluent sequences that are analogous to input sentences.
• Figure 5: More cases of embedding inversion attacks.