Simple Yet Effective: Extracting Private Data Across Clients in Federated Fine-Tuning of Large Language Models

TL;DR

This work proposes three simple yet effective extraction strategies that leverage contextual prefixes from the attacker's local data, including frequency-based prefix sampling and local fine-tuning to amplify memorization in order to recover personally identifiable information memorized from other clients'data.

Abstract

Federated large language models (FedLLMs) enable cross-silo collaborative training among institutions while preserving data locality, making them appealing for privacy-sensitive domains such as law, finance, and healthcare. However, the memorization behavior of LLMs can lead to privacy risks that may cause cross-client data leakage. In this work, we study the threat of cross-client data extraction, where a semi-honest participant attempts to recover personally identifiable information (PII) memorized from other clients' data. We propose three simple yet effective extraction strategies that leverage contextual prefixes from the attacker's local data, including frequency-based prefix sampling and local fine-tuning to amplify memorization. To evaluate these attacks, we construct a Chinese legal-domain dataset with fine-grained PII annotations consistent with CPIS, GDPR, and CCPA standards, and assess extraction performance using two metrics: coverage and efficiency. Experimental results show that our methods can recover up to 56.6% of victim-exclusive PII, where names, addresses, and birthdays are particularly vulnerable. These findings highlight concrete privacy risks in FedLLMs and establish a benchmark and evaluation framework for future research on privacy-preserving federated learning. Code and data are available at https://github.com/SMILELab-FL/FedPII.

Figures (14)

• Figure 1: Overview of cross-silo FedLLMs and the proposed privacy attack. In cross-silo FL, institutions such as banks, courts, and hospitals collaboratively fine-tune a shared model under the coordination of a central server, keeping data local. A semi-honest client leverages its local data to construct PII-related prefixes and queries the aggregated global FedLLM, leading to cross-client data leakage. The proposed strategies—Contextual Prefix Sampling, Frequency-prioritized (FP) Sampling, and Latent Association Fine-tuning (LAFt)—achieve up to 56.6% recovery of victim-exclusive PII, with names, addresses, and birthdays being the most vulnerable categories.
• Figure 2: Distribution of de-duplicated PII instances by label category.
• Figure 3: Label distribution of deduplicated victim-exclusive PII extracted by Qwen1-8B (without LAFt, using prefix set $P_c$). Results for Baichuan2-7B are shown in Appendix Figure \ref{['fig:VxPII_distribution_Baichuan2']}.
• Figure 4: Coverage rate (CR) and efficiency (EF) under varying prefix budgets $B$ for prefix sets $P_c$ and $P_{f \geq 1}$. Prefix set $P_{f \geq 1}$ is frequency-sorted in descending order (see Section \ref{['sec:fp_prefix_sampling']}). Budget values are scaled exponentially (base 10); model used is Qwen1-8B.
• Figure 5: VxPII counts under varying prefix budgets ($B$) for prefix sets $P_c$ and $P_{f \geq 1}$. Prefix set $P_{f \geq 1}$ is frequency-sorted in descending order (see Section \ref{['sec:fp_prefix_sampling']}) and truncated to match the size of $P_c$ here.

Theorems & Definitions (3)

• Definition 1: Extracted
• Definition 2: Coverage Rate
• Definition 3: Efficiency