Personalized Federated Training of Diffusion Models with Privacy Guarantees

TL;DR

This paper tackles data scarcity and privacy constraints in sensitive domains by proposing PFDM, a personalized federated diffusion framework that trains diffusion models without sharing raw data. PFDM decomposes the reverse diffusion into client-specific and global denoisers, enabling clients to control fine-grained generation while the global denoiser learns from noisy, diffused data to preserve privacy. A formal local differential privacy guarantee is established for the global denoiser, with the privacy-utility trade-off tuned by the diffusion steps, and per-pixel DP variants are discussed. Empirical results on CIFAR-10 and MNIST demonstrate competitive performance compared to centralized training and clear improvements over non-collaborative baselines, especially under high data heterogeneity and when generating biased-prone minority classes, highlighting the method’s practical impact for private, collaborative data synthesis.

Abstract

The scarcity of accessible, compliant, and ethically sourced data presents a considerable challenge to the adoption of artificial intelligence (AI) in sensitive fields like healthcare, finance, and biomedical research. Furthermore, access to unrestricted public datasets is increasingly constrained due to rising concerns over privacy, copyright, and competition. Synthetic data has emerged as a promising alternative, and diffusion models -- a cutting-edge generative AI technology -- provide an effective solution for generating high-quality and diverse synthetic data. In this paper, we introduce a novel federated learning framework for training diffusion models on decentralized private datasets. Our framework leverages personalization and the inherent noise in the forward diffusion process to produce high-quality samples while ensuring robust differential privacy guarantees. Our experiments show that our framework outperforms non-collaborative training methods, particularly in settings with high data heterogeneity, and effectively reduces biases and imbalances in synthetic data, resulting in fairer downstream models.

Figures (4)

• Figure 1: The effect of data heterogeneity on generation. We divide the MNIST dataset into two clusters, each containing five classes. We then construct two distinct datasets, each comprising 5,000 samples from one cluster and 50 from the other, with the majority-minority clusters reversed between datasets. Our goal is to perform conditional image generation for all classes. We explore two training approaches: (1) training a single diffusion model independently on each dataset (second row) and (2) using our personalized framework, which trains two models per dataset---a shared global model and a personalized local model (first row). We observe that models trained without collaboration perform significantly worse on the minority classes and often generate majority-class images even when prompted with a minority-class label. For example, the model confuses a 4 with a 7 in the bottom-left figure. In contrast, the personalized framework produces higher-quality images for the minority classes, demonstrating that the global model captures and transfers shared features across datasets. We discuss privacy guarantees in Section \ref{['sec:result']}.
• Figure 2: Illustration of generated samples using shared global denoiser in Figure \ref{['fig:MNIST']}.
• Figure 3: CIFAR-10 samples generated by different methods. (a), (b) correspond to samples generated for classes airplane, car, ship, truck. (c), (d) correspond to samples generated for classes deer, dog, frog, horse. We report the results for the model trained with cluster 1 as majority class.
• Figure 4: Illustration of our sampling procedure. The global step 0 correspond to the output using the shared global denoiser. The personal steps correspond to the outputs of using personalized denoiser.

Theorems & Definitions (7)

• Definition 3.1: $(\epsilon,\delta)$-DP
• Definition 3.2: $(\epsilon,\delta)$-LDP
• Theorem 5.1: Privacy Guarantee of PFDM
• Definition B.1: RDP
• Lemma B.2: RDP to DP
• Lemma B.3: Gaussian Mechanism
• proof : Proof of Theorem \ref{['thm:privacy']}