Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Differentially Private Gradient Flow based on the Sliced Wasserstein Distance

Ilana Sebag, Muni Sreenivas Pydi, Jean-Yves Franceschi, Alain Rakotomamonjy, Mike Gartrell, Jamal Atif, Alexandre Allauzen

TL;DR

This paper tackles privacy in generative modeling by formulating a differentially private gradient flow on the space of probability measures, driven by the Gaussian-smoothed Sliced Wasserstein Distance and its stochastic differential equation. It defines a functional ${\cal F}_{\lambda,\sigma}^\nu(\mu)$, proves the existence of a generalized minimizing movement scheme, and derives discretized particle updates whose drift incorporates a Gaussian mechanism, enabling DP guarantees from both smoothing and diffusion terms. The authors present two DP implementations (DPSWflow-r with projection resampling and DPSWflow with pre-sampled projections), analyze privacy amplification and budget tracking, and empirically show superior data fidelity compared to a DP generator baseline on MNIST, FashionMNIST, and CelebA under multiple privacy budgets. This approach provides a principled, DP-compliant alternative to DP-SGD for generative modeling, with clear pathways for tracking privacy and improving utility at fixed privacy levels.

Abstract

Safeguarding privacy in sensitive training data is paramount, particularly in the context of generative modeling. This can be achieved through either differentially private stochastic gradient descent or a differentially private metric for training models or generators. In this paper, we introduce a novel differentially private generative modeling approach based on a gradient flow in the space of probability measures. To this end, we define the gradient flow of the Gaussian-smoothed Sliced Wasserstein Distance, including the associated stochastic differential equation (SDE). By discretizing and defining a numerical scheme for solving this SDE, we demonstrate the link between smoothing and differential privacy based on a Gaussian mechanism, due to a specific form of the SDE's drift term. We then analyze the differential privacy guarantee of our gradient flow, which accounts for both the smoothing and the Wiener process introduced by the SDE itself. Experiments show that our proposed model can generate higher-fidelity data at a low privacy budget compared to a generator-based model, offering a promising alternative.

Differentially Private Gradient Flow based on the Sliced Wasserstein Distance

TL;DR

This paper tackles privacy in generative modeling by formulating a differentially private gradient flow on the space of probability measures, driven by the Gaussian-smoothed Sliced Wasserstein Distance and its stochastic differential equation. It defines a functional , proves the existence of a generalized minimizing movement scheme, and derives discretized particle updates whose drift incorporates a Gaussian mechanism, enabling DP guarantees from both smoothing and diffusion terms. The authors present two DP implementations (DPSWflow-r with projection resampling and DPSWflow with pre-sampled projections), analyze privacy amplification and budget tracking, and empirically show superior data fidelity compared to a DP generator baseline on MNIST, FashionMNIST, and CelebA under multiple privacy budgets. This approach provides a principled, DP-compliant alternative to DP-SGD for generative modeling, with clear pathways for tracking privacy and improving utility at fixed privacy levels.

Abstract

Safeguarding privacy in sensitive training data is paramount, particularly in the context of generative modeling. This can be achieved through either differentially private stochastic gradient descent or a differentially private metric for training models or generators. In this paper, we introduce a novel differentially private generative modeling approach based on a gradient flow in the space of probability measures. To this end, we define the gradient flow of the Gaussian-smoothed Sliced Wasserstein Distance, including the associated stochastic differential equation (SDE). By discretizing and defining a numerical scheme for solving this SDE, we demonstrate the link between smoothing and differential privacy based on a Gaussian mechanism, due to a specific form of the SDE's drift term. We then analyze the differential privacy guarantee of our gradient flow, which accounts for both the smoothing and the Wiener process introduced by the SDE itself. Experiments show that our proposed model can generate higher-fidelity data at a low privacy budget compared to a generator-based model, offering a promising alternative.
Paper Structure (27 sections, 14 theorems, 63 equations, 4 figures, 1 table, 2 algorithms)

This paper contains 27 sections, 14 theorems, 63 equations, 4 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Sliced Wasserstein Distance
  4. Wasserstein Gradient Flows
  5. Differential Privacy
  6. Differentially Private Gradient Flow
  7. Gradient Flow of the Smoothed Sliced Wasserstein Distance
  8. Discretized and Private Gradient Flow Algorithms
  9. Privacy Guarantee
  10. Privacy guarantee arising from the random projection
  11. Privacy guarantee arising from the diffusion term
  12. On the impact of (re-)sampling on privacy
  13. Experiments
  14. Toy Problem
  15. Comparisons and Baselines
  16. ...and 12 more sections

Key Result

Theorem 1

Let $\nu\in {\cal P} (\overline{B}(0,r))$ have a strictly posi-tive smooth density. For $\lambda>0$ and $r > \sqrt{d}$, let the starting distribution $\mu_0\in {\cal P} (\overline{B}(0,r))$ have a density $\rho_0\in \mathrm{L}^\infty(\overline{B}(0,1))$. There exists a minimizing movement scheme $( with: Here, $\psi_{\mu_t, \theta}^{(\sigma)}$ is the Kantorovich potential (see Section wassbackgr

Figures (4)

  • Figure 1: Examples of particle flows for (top) sliced Wasserstein flow, (middle) our DPSWflow with $\sigma=0.5$, and (bottom) $\sigma = 1$. Each panel shows the level sets (in black) of the target distribution, which is composed of $5$ Gaussians, as well as the particles (in green). The columns depict the particles after the (left) first step, (middle) $10$-th, and (right) the $200$-th steps of the flow.
  • Figure 2: Generated images from DPSWflow-r (upper row) and DPSWgen (lower row) for MNIST, FashionMNIST, and Celeba with no DP: $\varepsilon=\infty$.
  • Figure 3: Generated images from DPSWflow-r (upper row) and DPSWgen (lower row) for MNIST, FashionMNIST, and Celeba with DP: $\varepsilon=10$.
  • Figure 4: Generated images from DPSWflow-r (upper row) and DPSWgen (lower row) for MNIST, FashionMNIST, and Celeba with DP: $\varepsilon=5$.

Theorems & Definitions (25)

  • Definition 1
  • Theorem 1
  • proof : Proof sketch
  • Proposition 1
  • Theorem 2
  • Lemma 1: rakotomamonjy2021differentially
  • Lemma 2: balle2019privacy
  • Theorem 3
  • proof
  • Proposition 2
  • ...and 15 more