Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Privacy without Noisy Gradients: Slicing Mechanism for Generative Model Training

Kristjan Greenewald, Yuancheng Yu, Hao Wang, Kai Xu

TL;DR

This work considers the slicing privacy mechanism that injects noise into random low-dimensional projections of the private data, and provides strong privacy guarantees for it, and introduces the smoothed-sliced $f$-divergence and shows it enjoys statistical consistency.

Abstract

Training generative models with differential privacy (DP) typically involves injecting noise into gradient updates or adapting the discriminator's training procedure. As a result, such approaches often struggle with hyper-parameter tuning and convergence. We consider the slicing privacy mechanism that injects noise into random low-dimensional projections of the private data, and provide strong privacy guarantees for it. These noisy projections are used for training generative models. To enable optimizing generative models using this DP approach, we introduce the smoothed-sliced $f$-divergence and show it enjoys statistical consistency. Moreover, we present a kernel-based estimator for this divergence, circumventing the need for adversarial training. Extensive numerical experiments demonstrate that our approach can generate synthetic data of higher quality compared with baselines. Beyond performance improvement, our method, by sidestepping the need for noisy gradients, offers data scientists the flexibility to adjust generator architecture and hyper-parameters, run the optimization over any number of epochs, and even restart the optimization process -- all without incurring additional privacy costs.

Privacy without Noisy Gradients: Slicing Mechanism for Generative Model Training

TL;DR

This work considers the slicing privacy mechanism that injects noise into random low-dimensional projections of the private data, and provides strong privacy guarantees for it, and introduces the smoothed-sliced -divergence and shows it enjoys statistical consistency.

Abstract

Training generative models with differential privacy (DP) typically involves injecting noise into gradient updates or adapting the discriminator's training procedure. As a result, such approaches often struggle with hyper-parameter tuning and convergence. We consider the slicing privacy mechanism that injects noise into random low-dimensional projections of the private data, and provide strong privacy guarantees for it. These noisy projections are used for training generative models. To enable optimizing generative models using this DP approach, we introduce the smoothed-sliced -divergence and show it enjoys statistical consistency. Moreover, we present a kernel-based estimator for this divergence, circumventing the need for adversarial training. Extensive numerical experiments demonstrate that our approach can generate synthetic data of higher quality compared with baselines. Beyond performance improvement, our method, by sidestepping the need for noisy gradients, offers data scientists the flexibility to adjust generator architecture and hyper-parameters, run the optimization over any number of epochs, and even restart the optimization process -- all without incurring additional privacy costs.

Paper Structure

This paper contains 24 sections, 7 theorems, 45 equations, 2 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries and Problem Setup
  3. Differential Privacy
  4. f-divergence and Moment Matching
  5. Main Results
  6. Smoothed-sliced f-divergence
  7. Slicing Privacy Mechanism
  8. Training Generative Models
  9. Numerical Experiments
  10. Synthetic Tabular Data
  11. Baselines.
  12. Data.
  13. Evaluation metrics.
  14. Main results and observations.
  15. Synthetic Image Data
  16. ...and 9 more sections

Key Result

Proposition 1

The smoothed-sliced $f$-divergence is non-negative: $\textnormal{SD}_{f,k,\sigma^2}(P_{\textnormal{X}}\|Q_{\textnormal{X}}) \geq 0$ for any $k \geq 1$ and $\sigma \geq 0$. If $f$ is strictly convex at $1$ and $P_{\textnormal{X}},Q_{\textnormal{X}}$ have moment generating functions, then $\textnormal

Figures (2)

  • Figure 1: We compare accuracy for downstream classification as a function of privacy budget ($\epsilon$) for synthetic MNIST data created by MERF with our Algorithm \ref{['alg:dp_Gen']} and SliceWass. Note that the two slicing-mechanism-based approaches outperform MERF for higher privacy budgets.
  • Figure 2: Unsupervised domain adaptation between from MNIST to USPS and vice versa.

Theorems & Definitions (21)

  • Definition 1: Dataset adjacency
  • Definition 2: $(\epsilon,\delta)$ differential privacy
  • Definition 3
  • Definition 4
  • Proposition 1
  • Definition 5
  • Remark 1
  • Theorem 1
  • Remark 2: Choosing $\alpha$
  • Corollary 1: Consistency
  • ...and 11 more