Federated Learning with Unlabeled Clients: Personalization Can Happen in Low Dimensions

TL;DR

This work tackles personalization in federated learning when many clients lack labels by introducing FLowDUP, which generates personalized models from unlabeled data through a hypernetwork that outputs low-dimensional subspace parameters: $\theta = \theta_0 + P v$ with $v = h(X;\psi_h)$. The training objective combines labeled-client feedback with a learnable regularizer $\Omega$ derived from unlabeled data, and its design is theoretically motivated by a transductive multi-task PAC-Bayes generalization bound that bounds the true risk using the labeled-training risk plus complexity terms. Empirically, FLowDUP achieves strong performance across heterogeneous datasets (CIFAR-10, Fashion-MNIST, FEMNIST) and benefits from unlabeled clients during training, with ablations clarifying the impact of subspace dimension, architecture choices, and dataset embeddings. The approach preserves federated privacy by keeping data on-device while enabling effective personalization, though it may struggle when conditional distributions vary in ways not captured by marginals alone, suggesting directions for incorporating additional side information. Overall, FLowDUP provides a scalable, principled pathway to personalization in FL with unlabeled clients, combining efficiency with theoretical guarantees and strong empirical results.

Abstract

Personalized federated learning has emerged as a popular approach to training on devices holding statistically heterogeneous data, known as clients. However, most existing approaches require a client to have labeled data for training or finetuning in order to obtain their own personalized model. In this paper we address this by proposing FLowDUP, a novel method that is able to generate a personalized model using only a forward pass with unlabeled data. The generated model parameters reside in a low-dimensional subspace, enabling efficient communication and computation. FLowDUP's learning objective is theoretically motivated by our new transductive multi-task PAC-Bayesian generalization bound, that provides performance guarantees for unlabeled clients. The objective is structured in such a way that it allows both clients with labeled data and clients with only unlabeled data to contribute to the training process. To supplement our theoretical results we carry out a thorough experimental evaluation of FLowDUP, demonstrating strong empirical performance on a range of datasets with differing sorts of statistically heterogeneous clients. Through numerous ablation studies, we test the efficacy of the individual components of the method.

Figures (1)

• Figure 1: Visualization of the client embeddings on Rotated Fashion-MNIST. Projection to two dimensions using PCA (left) and t-SNE (right).

Theorems & Definitions (5)

• Theorem 4.1
• Theorem A.1
• proof
• Theorem A.1
• proof