PRISM: Privacy-Preserving Improved Stochastic Masking for Federated Generative Models

TL;DR

PRISM is the first to successfully generate images under challenging non-IID and privacy-preserving FL environments on complex datasets, where previous methods have struggled, and yields a lightweight model without extra pruning or quantization, making it ideal for environments such as edge devices.

Abstract

Despite recent advancements in federated learning (FL), the integration of generative models into FL has been limited due to challenges such as high communication costs and unstable training in heterogeneous data environments. To address these issues, we propose PRISM, a FL framework tailored for generative models that ensures (i) stable performance in heterogeneous data distributions and (ii) resource efficiency in terms of communication cost and final model size. The key of our method is to search for an optimal stochastic binary mask for a random network rather than updating the model weights, identifying a sparse subnetwork with high generative performance; i.e., a ``strong lottery ticket''. By communicating binary masks in a stochastic manner, PRISM minimizes communication overhead. This approach, combined with the utilization of maximum mean discrepancy (MMD) loss and a mask-aware dynamic moving average aggregation method (MADA) on the server side, facilitates stable and strong generative capabilities by mitigating local divergence in FL scenarios. Moreover, thanks to its sparsifying characteristic, PRISM yields a lightweight model without extra pruning or quantization, making it ideal for environments such as edge devices. Experiments on MNIST, FMNIST, CelebA, and CIFAR10 demonstrate that PRISM outperforms existing methods, while maintaining privacy with minimal communication costs. PRISM is the first to successfully generate images under challenging non-IID and privacy-preserving FL environments on complex datasets, where previous methods have struggled.

Figures (10)

• Figure 1: Overview of PRISM. PRISM finds the supermask for generative models in a FL scenario. At every round $t$, each client $k$ updates a local score $s_t^k$ via MMD loss (Step 1) and generates the privacy-preserving binary mask $M_t^k$ (Step 2-1), which is sent to the server. The server aggregates the masks to obtain the global probability $\theta_{t+1}$ (Step 3), which is converted to a score $s_{t+1}$ and broadcasted to the clients for the next round (Step 4). The global probability $\theta_{t+1}$ is gradually updated based on mask correlation $\lambda$ between $M_t^g$ and $M_{t-1}^g$.
• Figure 2: Qualitative results in IID scenario with a privacy budget $(\epsilon, \delta)=(9.8, 10^{-5})$. We compare generated images from the models in Table \ref{['tab:iid_dp']} on MNIST, FMNIST, and CelebA. $\dagger$ indicates that MADA is removed.
• Figure 3: Qualitative results in Non-IID scenario with a privacy budget $(\epsilon, \delta)=(9.8, 10^{-5})$. We compare generated images from the models in Table \ref{['tab:noniid_dp']} on MNIST, FMNIST, and CelebA. $\dagger$ indicates that MADA is removed.
• Figure 4: The performance of baselines and our PRISM with privacy budget $(\epsilon, \delta)=(9.8, 10^{-5})$. X-axis represents the number of parameters of final generator, while Y-axis represents FID. The diameter of each circle denotes the required communication cost at every round. The ideal case is the bottom-left corner.
• Figure 5: Qualitative results in non-IID scenario without considering privacy budget. Generated images from the models in Table \ref{['tab:noniid']} on MNIST, FMNIST, CelebA, and CIFAR10. Here, we set $\alpha=80$ for PRISM$^{\ast}$.

Theorems & Definitions (4)

• Definition 1: ($\epsilon, \delta$)-Differential Privacy dp
• Definition 2: ($\epsilon, \delta$)-Differential Privacy
• Definition 3: ($\alpha, \epsilon$) Rényi Differential Privacy
• Theorem 1