Understanding Privacy Risks of Embeddings Induced by Large Language Models

TL;DR

This paper investigates privacy risks of embeddings produced by large language models by treating an LLM as an attacker that reconstructs original text and infers sensitive attributes from embeddings generated by open-source targets. It uses GPT-2 variants as attack models to recover text and to predict attributes across diverse datasets, showing that larger attack models and more expressive embedding targets substantially raise reconstruction and inference accuracy, even for out-of-distribution data. Key findings include that in-distribution texts are easier to reconstruct, shorter texts are at higher risk due to fixed embedding sizes, and external proxy embeddings can enable attribute inference when the original text is brief. The work discusses mitigation strategies (e.g., not disclosing raw texts, lengthening inputs, and developing privacy-preserving embeddings) and acknowledges limitations, emphasizing the need for defense mechanisms as embedding-based information leakage becomes more pronounced in real-world deployments.

Abstract

Large language models (LLMs) show early signs of artificial general intelligence but struggle with hallucinations. One promising solution to mitigate these hallucinations is to store external knowledge as embeddings, aiding LLMs in retrieval-augmented generation. However, such a solution risks compromising privacy, as recent studies experimentally showed that the original text can be partially reconstructed from text embeddings by pre-trained language models. The significant advantage of LLMs over traditional pre-trained models may exacerbate these concerns. To this end, we investigate the effectiveness of reconstructing original knowledge and predicting entity attributes from these embeddings when LLMs are employed. Empirical findings indicate that LLMs significantly improve the accuracy of two evaluated tasks over those from pre-trained models, regardless of whether the texts are in-distribution or out-of-distribution. This underscores a heightened potential for LLMs to jeopardize user privacy, highlighting the negative consequences of their widespread use. We further discuss preliminary strategies to mitigate this risk.

Figures (5)

• Figure 1: The fine-tuning of the foundation attack model. Initially, the attacker queries the target embedding model to convert the collected text into text embeddings. To signify the completion of the embedding input, an EOS (End-of-Sentence) token is appended to the text embedding. Next, the attacker selects the pre-trained GPT2 model as the attack model and uses the collected text and corresponding text embeddings as a dataset to train the attack model. When a text embedding is input, the attack model is trained to sequentially reconstruct the related original text.
• Figure 2: The similarity between the evaluation datasets and the Wiki dataset v.s. the best reconstruction performance.
• Figure 3: Impact of disclosed original texts volume on text reconstruction accuracy. Each column represents a different target embedding model. The first and second rows represent the reconstruction performance concerning the BLEU-1 and ROUGE-1 metrics, respectively.
• Figure 4: Influence of the text length. Error bars represent the mean reconstruction accuracy with 95% confidence intervals obtained from 10 independent trials, and each column corresponds to a different target embedding model.
• Figure 5: The inference framework of sensitive attributes. The attacker employs the same embedding model to convert original text and candidate attributes into embeddings. The attacker then identifies the attribute that exhibits the highest cosine similarity between its embedding and text embedding as sensitive information of the original text.