Stealing Training Data from Large Language Models in Decentralized Training through Activation Inversion Attack

TL;DR

The paper addresses privacy risks in decentralized LLM training by introducing Activation Inversion Attack (AIA), which reconstructs private training data from transmitted activations without disrupting training. It employs a two-step approach: constructing a shadow dataset using a public pre-trained shadow model to train an attack network that maps activations to text, enabling reconstruction of victim data during fine-tuning. Across GPT2-XL, Bloom-7B1, and LLaMA3-8B and multiple public corpora, AIA achieves fluent text reconstruction and substantial privacy leakage of PII types, demonstrating strong practical risk. The findings underscore the need for defenses in decentralized training pipelines and highlight limitations related to attack-victim architectural alignment and transferability, while providing actionable metrics for evaluating privacy risks in such systems.

Abstract

Decentralized training has become a resource-efficient framework to democratize the training of large language models (LLMs). However, the privacy risks associated with this framework, particularly due to the potential inclusion of sensitive data in training datasets, remain unexplored. This paper identifies a novel and realistic attack surface: the privacy leakage from training data in decentralized training, and proposes \textit{activation inversion attack} (AIA) for the first time. AIA first constructs a shadow dataset comprising text labels and corresponding activations using public datasets. Leveraging this dataset, an attack model can be trained to reconstruct the training data from activations in victim decentralized training. We conduct extensive experiments on various LLMs and publicly available datasets to demonstrate the susceptibility of decentralized training to AIA. These findings highlight the urgent need to enhance security measures in decentralized training to mitigate privacy risks in training LLMs.

Figures (7)

• Figure 1: Cosine similarity between activations for the same data in the pre-trained model and the fine-tuned model across layer index.
• Figure 2: Overview of Activation Inversion Attack (AIA). In a decentralized training system, the victim model $M_{\text{vic}}$ undergoes fine-tuning using private data $\mathcal{D}_{\text{vic}}$, which may contain personally identifiable information values (highlighted in yellow). An honest-but-curious attacker controlling the $i_{\text{att}}$-th stage of the pipeline: (1) records intermediate activation values $\bm a_{i_\text{att}-1}^{(t)}$ captured during the training process, and (2) collects shadow activations $\mathcal{D}_{\text{sha}}$ from the shadow model $M_{\text{sha}}$ to train the attack model $M_{\text{att}}$. Finally, the attacker uses $M_{\text{att}}$ to reconstruct the private data $\mathcal{D}_{\text{vic}}$, with the red and purple text representing precisely recovered and mostly recovered PII data, respectively.
• Figure 3: An example of PII data and baseline attacks. The private data includes information such as names, phone numbers, and email addresses. The True-Prefix attack leverages other private attributes to prompt the model to generate the target private attribute, while the SPT attack employs a trained soft prompt added before the query template to extract private information.
• Figure 4: Three comparative examples of generated texts versus original data. The yellow text represents the original PII data, while the red and purple texts represent precisely recovered and mostly recovered PII data, respectively. The text recovery performance improves from left to right.
• Figure 5: The attack performance of AIA on GPT2-XL and Bloom-7B1 models as the attacker's decoder layer index varies, with the attack performance generally decreasing as the layer index increases.