PaDPaF: Partial Disentanglement with Partially-Federated GANs

TL;DR

PaDPaF tackles learning personalized generative models under data heterogeneity by partially federating GAN components to separate content (global) from style (local). It introduces a partial-federation architecture with a global content encoder/discriminator and client-specific style modules, trained via FedAvg-style updates, and a latent-contrastive regularizer inspired by Barlow Twins to enforce content–style disentanglement, ensuring a robust, client-invariant content representation. The framework enables high-accuracy content classification with a simple linear head and the generation of locally unseen labels, while supporting data anonymization by sharing only content. Empirical evaluations on MNIST, CIFAR-10, and CelebA show that content features carry the majority of label information and that PaDPaF outperforms baselines in both generation and style transfer tasks, with potential extensions to VAEs and broader modalities.

Abstract

Federated learning has become a popular machine learning paradigm with many potential real-life applications, including recommendation systems, the Internet of Things (IoT), healthcare, and self-driving cars. Though most current applications focus on classification-based tasks, learning personalized generative models remains largely unexplored, and their benefits in the heterogeneous setting still need to be better understood. This work proposes a novel architecture combining global client-agnostic and local client-specific generative models. We show that using standard techniques for training federated models, our proposed model achieves privacy and personalization by implicitly disentangling the globally consistent representation (i.e. content) from the client-dependent variations (i.e. style). Using such decomposition, personalized models can generate locally unseen labels while preserving the given style of the client and can predict the labels for all clients with high accuracy by training a simple linear classifier on the global content features. Furthermore, disentanglement enables other essential applications, such as data anonymization, by sharing only the content. Extensive experimental evaluation corroborates our findings, and we also discuss a theoretical motivation for the proposed approach.

Figures (21)

• Figure 1: The architecture of our model. Federated modules are aggregated in each communication round (shown in green, solid blocks), whereas private modules are not (shown in red, dashed blocks). The discriminative loss is just the GAN loss as described in \ref{['sec:fed-cgan']}, whereas the contrastive regularization term is explained in more detail in \ref{['sec:contrast-reg']}. We describe the model's components in full detail in the Appendix.
• Figure 2: A causal model with the generative latent variables $\mathbf{z}^c$ and $\mathbf{z}^s$ of $\mathbf{x}$. The dotted arrows show the underlying causal model without latents. Both latents depend on the client $i$, and we assume that the content latent $\mathbf{z}^c$ generates the label $\mathbf{y}$ as well. The blue arrows drop when we $do(\mathbf{z}^c)$, and the red arrows drop when we $do(\mathbf{z}^s)$. The dashed arrows illustrate the reduced influence of $i$ over the mechanism $i \to \mathbf{z}^c$ as we run federated averaging. $\mathbf{z}^s$ and its mechanism $i \to \mathbf{z}^c$, on the other hand, remain specific to the client in our case.
• Figure 3: Our partially-federated linear generator can learn the right data-generating process for this dataset (top right). Federated averaging on a linear model will yield the ordinary least squares solution at best (bottom left), whereas a linear model with a private bias has better generalization per client (bottom right). The top left plot shows the full dataset. (color = client)
• Figure 4: Left two plots: Accuracy and loss of a linear classifier on the content and style features of unsupervised GAN training on MNIST (i.e. labels $\mathbf{y}$ are not used in training). Right two plots: The progress of the accuracy and loss achieved by a linear classifier in terms of communication rounds (better seen with colors).
• Figure 5: Accuracy and loss of a linear classifier on the content and style features of unsupervised GAN training on CIFAR-10 (showing performance on the style features of one client only)