Learning Differentially Private Diffusion Models via Stochastic Adversarial Distillation

TL;DR

Privacy is a major barrier to releasing synthetic data from sensitive domains. DP-SAD introduces a three-component diffusion framework with a private teacher, a private student, and a discriminator that are trained through a combination of adversarial and stochastic distillation steps, leveraging diffusion time steps $T$ to dilute DP noise. It provides a Rényi differential privacy analysis and uses a Gaussian mechanism with gradient clipping to establish DP guarantees, achieving high utility under DP budgets. Empirically, DP-SAD outperforms 11 baselines on MNIST, FMNIST, and CelebA in terms of perceptual metrics and downstream classifier performance, demonstrating effective private diffusion-based data generation with practical training efficiency.

Abstract

While the success of deep learning relies on large amounts of training datasets, data is often limited in privacy-sensitive domains. To address this challenge, generative model learning with differential privacy has emerged as a solution to train private generative models for desensitized data generation. However, the quality of the images generated by existing methods is limited due to the complexity of modeling data distribution. We build on the success of diffusion models and introduce DP-SAD, which trains a private diffusion model by a stochastic adversarial distillation method. Specifically, we first train a diffusion model as a teacher and then train a student by distillation, in which we achieve differential privacy by adding noise to the gradients from other models to the student. For better generation quality, we introduce a discriminator to distinguish whether an image is from the teacher or the student, which forms the adversarial training. Extensive experiments and analysis clearly demonstrate the effectiveness of our proposed method.

Figures (10)

• Figure 1: Overview of our DP-SAD. We first train a teacher model $\psi$ using the private data without protection. After that, we train a student model with the private data and the fixed teacher model in a distillation manner. In addition, we add a discriminator and view the student model as a generator to form adversarial training. Finally, for privacy, we achieve differential privacy by clipping with bound $C$ and adding noise $\mathcal{N}$ to the gradients during backpropagation. Furthermore, we accelerate the training by using the gradient of a random time step $CLIP(g^r,C)$ instead of averaging the gradients of all time steps in the reaction process $\frac{1}{T}\sum_{t=1}^TCLIP(g^t,C)$, where $CLIP(*,C)=*/max(1,\frac{||*||_2}{C})$.
• Figure 2: Left: Illustration of the proposed discriminator. We concatenate the outputs of the teacher and student models, utilizing these combined outputs as the inputs for the discriminator. The discriminator distinguishes whether the input image originates from the teacher model or the student model. Right: Use the gradients of step $r$ as a substitute for the average gradients over $T$ steps. By obtaining $x_r$ through the forward process, we prevent the teacher from inferring from noise to $x_r$, thereby saving computational time.
• Figure 3: Visualization results of DP-GAN, GS-WGAN, DP-MERF, P3GM, DataLens, DPGEN, DP-LDM and our DP-SAD on CelebA at 32$\times$32 and 64$\times$64 resolutions.
• Figure 4: Generated samples by DP-SAD on CelebA-G and CelebA-H under different privacy budget ($\varepsilon=1$ and $\varepsilon=10$).
• Figure 5: Left: Perceptual scores on CelebA under different time steps. Right: Perceptual scores on CelebA under different model conditioning settings (w/o: without model conditioning, w/s: with student conditioning, w/d: with discriminator conditioning, w/b: with both model conditioning).

Theorems & Definitions (13)

• definition thmcounterdefinition: Differential Privacy
• theorem thmcountertheorem: Post-processing
• definition thmcounterdefinition: R$\acute{\textbf{e}}$nyi Differential Privacy
• theorem thmcountertheorem: Convert RDP to DP
• theorem thmcountertheorem: Gaussian Mechanism
• lemma thmcounterlemma
• proof
• theorem thmcountertheorem
• proof
• lemma thmcounterlemma