Little is Enough: Boosting Privacy by Sharing Only Hard Labels in Federated Semi-Supervised Learning
Amr Abourayya, Jens Kleesiek, Kanishka Rao, Erman Ayday, Bharat Rao, Geoff Webb, Michael Kamp
TL;DR
This work tackles privacy in federated learning by moving beyond parameter and soft-label sharing to a federated co-training approach that exchanges only hard labels on a public unlabeled dataset. FedCT builds a consensus pseudo-labels mechanism (majority vote) enabling local models—potentially non-gradient-based like decision trees or random forests—to train collaboratively without exposing private data. Theoretical results establish convergence and provide a DP-ready bound via an XOR-based mechanism, yielding strong privacy with minimal utility loss, and empirical results show competitive accuracy across benchmarks, improvements for LLM fine-tuning, reduced communication, and applicability to interpretable models. Overall, FedCT offers a practical, privacy-preserving alternative for distributed sensitive domains (e.g., healthcare) where centralization or gradient-based aggregation is undesirable or impractical.
Abstract
In many critical applications, sensitive data is inherently distributed and cannot be centralized due to privacy concerns. A wide range of federated learning approaches have been proposed to train models locally at each client without sharing their sensitive data, typically by exchanging model parameters, or probabilistic predictions (soft labels) on a public dataset or a combination of both. However, these methods still disclose private information and restrict local models to those that can be trained using gradient-based methods. We propose a federated co-training (FedCT) approach that improves privacy by sharing only definitive (hard) labels on a public unlabeled dataset. Clients use a consensus of these shared labels as pseudo-labels for local training. This federated co-training approach empirically enhances privacy without compromising model quality. In addition, it allows the use of local models that are not suitable for parameter aggregation in traditional federated learning, such as gradient-boosted decision trees, rule ensembles, and random forests. Furthermore, we observe that FedCT performs effectively in federated fine-tuning of large language models, where its pseudo-labeling mechanism is particularly beneficial. Empirical evaluations and theoretical analyses suggest its applicability across a range of federated learning scenarios.