Leveraging Soft Prompts for Privacy Attacks in Federated Prompt Tuning

TL;DR

PromptMIA reveals a new privacy vulnerability in federated prompt tuning by injecting adversarial soft prompts into the global prompt pool and monitoring which prompts are updated to infer a target sample's membership in a client's data. The attack relies on a structured key-prompt selection mechanism and a security-game formulation, achieving near-perfect true positive rates and strong overall advantage across diverse datasets and backbones. The authors provide a formal lower bound on attack advantage and demonstrate that standard defenses, including DPSGD, input noise, and classical anomaly detectors, offer limited protection in this setting. Empirical results across seven datasets and three vision transformers, complemented by theoretical analysis, underscore the need for defense strategies specifically designed for federated prompt-tuning paradigms. The work also discusses potential mitigations, such as calibrated alignment controls, but highlights substantial privacy-utility trade-offs and practical challenges. Reproducibility resources will be released to support further examination of prompt-based privacy risks in FL.

Abstract

Membership inference attack (MIA) poses a significant privacy threat in federated learning (FL) as it allows adversaries to determine whether a client's private dataset contains a specific data sample. While defenses against membership inference attacks in standard FL have been well studied, the recent shift toward federated fine-tuning has introduced new, largely unexplored attack surfaces. To highlight this vulnerability in the emerging FL paradigm, we demonstrate that federated prompt-tuning, which adapts pre-trained models with small input prefixes to improve efficiency, also exposes a new vector for privacy attacks. We propose PromptMIA, a membership inference attack tailored to federated prompt-tuning, in which a malicious server can insert adversarially crafted prompts and monitors their updates during collaborative training to accurately determine whether a target data point is in a client's private dataset. We formalize this threat as a security game and empirically show that PromptMIA consistently attains high advantage in this game across diverse benchmark datasets. Our theoretical analysis further establishes a lower bound on the attack's advantage which explains and supports the consistently high advantage observed in our empirical results. We also investigate the effectiveness of standard membership inference defenses originally developed for gradient or output based attacks and analyze their interaction with the distinct threat landscape posed by PromptMIA. The results highlight non-trivial challenges for current defenses and offer insights into their limitations, underscoring the need for defense strategies that are specifically tailored to prompt-tuning in federated settings.

Figures (20)

• Figure 1: PromptMIA workflow: (1) the server injects adversarial prompts designed for a target sample into the global pool; (2) the modified pool is broadcast to clients; (3) each client performs query-key matching, which selects all adversarial prompts if the target sample is present; (4) selected prompts are locally updated and (5) returned to the server. By monitoring which prompts are updated, the server infers the target’s membership in client data.
• Figure 2: Comparison of the global key distributions produced by a ViT-B/32 model trained on CIFAR-10 after 60 global epochs, visualized using t-SNE. Blue keys are benign keys, and red keys are adversarial keys.
• Figure 3: Left: t-SNE projection of the global keys and query vectors from the train set. Center: K-Means clustering of keys with queries. Right: Each cluster modeled as a spherical Gaussian distribution centered at a key. All visualizations are generated from a ViT-B32 model trained on CIFAR-10 for 60 global epochs.
• Figure 4: Performance of PromptMIA vs Naive averaged across three models. Each subplot shows Advantage and Attack Success Rate w.r.t Batch Size across CIFAR10, CIFAR100, TinyImageNet, and FourDataset.
• Figure 5: Visualization of outlier detection methods on CIFAR-10 trained Vit-B32. Blue keys are benign keys. Red keys are adversarial keys. Crossed keys are flagged as outliers from the corresponding algorithm.

Theorems & Definitions (10)

• Theorem 1: True Positive Rate
• Lemma 1
• Theorem 2: False Positive Rate
• Corollary 1: Attack Advantage
• Theorem 1: True Positive Rate
• proof
• Lemma 1: Single-Point Flip Probability
• proof
• Theorem 2: FPR Bound for $N=1$
• proof