Differentially Private Diffusion Models Generate Useful Synthetic Images

TL;DR

This work tackles the challenge of producing useful, privacy-preserving synthetic images by scaling differentially private diffusion models. It leverages public pre-training (ImageNet) and targeted training strategies—augmentation multiplicity and biased timestep sampling—to privately fine-tune large diffusion models (>80M parameters) and achieve state-of-the-art downstream performance on CIFAR-10 (FID 9.8; accuracy ~88%) and Camelyon17 (91.1%). The authors also demonstrate the value of synthetic data for hyperparameter tuning and model selection, even under distribution shift between pre-training and fine-tuning data. Overall, the results show that DP diffusion can yield privately generated data that meaningfully supports downstream tasks, with practical implications for deploying privacy-preserving ML in sensitive domains.

Abstract

The ability to generate privacy-preserving synthetic versions of sensitive image datasets could unlock numerous ML applications currently constrained by data availability. Due to their astonishing image generation quality, diffusion models are a prime candidate for generating high-quality synthetic data. However, recent studies have found that, by default, the outputs of some diffusion models do not preserve training data privacy. By privately fine-tuning ImageNet pre-trained diffusion models with more than 80M parameters, we obtain SOTA results on CIFAR-10 and Camelyon17 in terms of both FID and the accuracy of downstream classifiers trained on synthetic data. We decrease the SOTA FID on CIFAR-10 from 26.2 to 9.8, and increase the accuracy from 51.0% to 88.0%. On synthetic data from Camelyon17, we achieve a downstream accuracy of 91.1% which is close to the SOTA of 96.5% when training on the real data. We leverage the ability of generative models to create infinite amounts of data to maximise the downstream prediction performance, and further show how to use synthetic data for hyperparameter tuning. Our results demonstrate that diffusion models fine-tuned with differential privacy can produce useful and provably private synthetic data, even in applications with significant distribution shift between the pre-training and fine-tuning distributions.

Figures (11)

• Figure 1: DP diffusion models are capable of producing high-quality images. More images can be found in Figures \ref{['fig:mnist']}, \ref{['fig:camelyon']}, \ref{['fig:cifar']}.
• Figure 2: FID on CIFAR-10 for different privacy budgets. Our performance at $\epsilon=5$ beats the SOTA, when pre-training on ImageNet, for $\epsilon=\infty$. Results for harder2022differentially are taken from their paper.
• Figure 3: Downstream Top-1 accuracy of a CIFAR-10 WRN-40-4 as function of the number of synthetic data samples used to train it. The accuracy increases considerably as a function of the dataset size.
• Figure 4: We observe that models rank similarly when evaluated on synthetic and real data. This suggests that findings on hyperparameter selection made on synthetic data can be transferred to the private dataset.
• Figure 5: Random samples drawn from a DP image diffusion model trained on MNIST.

Theorems & Definitions (1)

• Definition 3.1: Differential Privacy dwork2006calibrating