Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PrivImage: Differentially Private Synthetic Image Generation using Diffusion Models with Semantic-Aware Pretraining

Kecen Li, Chen Gong, Zhixiang Li, Yuzhong Zhao, Xinwen Hou, Tianhao Wang

TL;DR

A novel DP image synthesis method, termed PRIVIMAGE, which meticulously selects pre-training data, promoting the efficient creation of DP datasets with high fidelity and utility and achieves superior synthetic performance and conserves more computational resources.

Abstract

Differential Privacy (DP) image data synthesis, which leverages the DP technique to generate synthetic data to replace the sensitive data, allowing organizations to share and utilize synthetic images without privacy concerns. Previous methods incorporate the advanced techniques of generative models and pre-training on a public dataset to produce exceptional DP image data, but suffer from problems of unstable training and massive computational resource demands. This paper proposes a novel DP image synthesis method, termed PRIVIMAGE, which meticulously selects pre-training data, promoting the efficient creation of DP datasets with high fidelity and utility. PRIVIMAGE first establishes a semantic query function using a public dataset. Then, this function assists in querying the semantic distribution of the sensitive dataset, facilitating the selection of data from the public dataset with analogous semantics for pre-training. Finally, we pre-train an image generative model using the selected data and then fine-tune this model on the sensitive dataset using Differentially Private Stochastic Gradient Descent (DP-SGD). PRIVIMAGE allows us to train a lightly parameterized generative model, reducing the noise in the gradient during DP-SGD training and enhancing training stability. Extensive experiments demonstrate that PRIVIMAGE uses only 1% of the public dataset for pre-training and 7.6% of the parameters in the generative model compared to the state-of-the-art method, whereas achieves superior synthetic performance and conserves more computational resources. On average, PRIVIMAGE achieves 30.1% lower FID and 12.6% higher Classification Accuracy than the state-of-the-art method. The replication package and datasets can be accessed online.

PrivImage: Differentially Private Synthetic Image Generation using Diffusion Models with Semantic-Aware Pretraining

TL;DR

A novel DP image synthesis method, termed PRIVIMAGE, which meticulously selects pre-training data, promoting the efficient creation of DP datasets with high fidelity and utility and achieves superior synthetic performance and conserves more computational resources.

Abstract

Differential Privacy (DP) image data synthesis, which leverages the DP technique to generate synthetic data to replace the sensitive data, allowing organizations to share and utilize synthetic images without privacy concerns. Previous methods incorporate the advanced techniques of generative models and pre-training on a public dataset to produce exceptional DP image data, but suffer from problems of unstable training and massive computational resource demands. This paper proposes a novel DP image synthesis method, termed PRIVIMAGE, which meticulously selects pre-training data, promoting the efficient creation of DP datasets with high fidelity and utility. PRIVIMAGE first establishes a semantic query function using a public dataset. Then, this function assists in querying the semantic distribution of the sensitive dataset, facilitating the selection of data from the public dataset with analogous semantics for pre-training. Finally, we pre-train an image generative model using the selected data and then fine-tune this model on the sensitive dataset using Differentially Private Stochastic Gradient Descent (DP-SGD). PRIVIMAGE allows us to train a lightly parameterized generative model, reducing the noise in the gradient during DP-SGD training and enhancing training stability. Extensive experiments demonstrate that PRIVIMAGE uses only 1% of the public dataset for pre-training and 7.6% of the parameters in the generative model compared to the state-of-the-art method, whereas achieves superior synthetic performance and conserves more computational resources. On average, PRIVIMAGE achieves 30.1% lower FID and 12.6% higher Classification Accuracy than the state-of-the-art method. The replication package and datasets can be accessed online.
Paper Structure (36 sections, 5 theorems, 12 equations, 17 figures, 9 tables, 2 algorithms)

This paper contains 36 sections, 5 theorems, 12 equations, 17 figures, 9 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Differential Privacy
  4. Image Generative Models
  5. DP Image Dataset Synthesis
  6. Methodology
  7. Motivation
  8. Query Image Semantics
  9. Semantic Distribution Query
  10. Pre-training and Fine-tuning
  11. Experiment Setup
  12. Investigated Datasets
  13. Baselines
  14. Evaluation Metrics
  15. Implementation
  16. ...and 21 more sections

Key Result

Theorem 2.1

(RDP for SGM sgm) Let $p_0$ and $p_1$ denote the PDF of $\mathcal{N}(0,\sigma^2)$ and $\mathcal{N}(1,\sigma^2)$ respectively. A $SG{M_{M,q,\sigma }}\left( D \right)$ satisfies ($\alpha, \gamma$)-RDP for any $\gamma$ such that,

Figures (17)

  • Figure 1: Two images with their captions. The same and different semantics two images own are in green and red respectively. Although the two images differ a lot in pixel, they have the same semantics, man and dog.
  • Figure 2: An example of using the semantic query function to retrieve the semantic distribution from the sensitive dataset. We first train the semantic query function using the public dataset. This function is used to obtain the semantic distribution of the sensitive dataset. To ensure privacy, we then incorporate the Gaussian noise into our query results.
  • Figure 3: The t-SNE visualizations depict the embedding distribution of the synthetic images using four DP dataset synthetic methods within the CIFAR-10 dataset with $\varepsilon=10$. We obtain embeddings by using CNN classifiers trained on synthetic images.
  • Figure 4: Examples of Synthetic CIFAR-10cifar10 images with $\varepsilon=10$. These generative models are trained using our tools Privimage+D, as well as PDP-Diffusion dpdm-sota and DPGAN-p. Each row corresponds to a category from the CIFAR-10 dataset.
  • Figure 5: (Left) The FID of baselines pre-trained on the entire public dataset and ours Privimage pre-trained on the selected public dataset. (Right) The SDS between the sensitive dataset and different pre-training datasets.
  • ...and 12 more figures

Theorems & Definitions (7)

  • Definition 2.1
  • Definition 2.2
  • Theorem 2.1
  • Theorem 3.1
  • Theorem F.1
  • Theorem F.2
  • Theorem F.3