Wasserstein Differential Privacy

TL;DR

This work introduces Wasserstein differential privacy (WDP), a metric-based DP framework that uses $W_\mu$ to bound privacy loss and ensures symmetry, triangle inequality, and non-negativity. It derives a suite of properties, advanced composition, and a Wasserstein accountant for tracking privacy budgets under subsampling, facilitating DP-SGD in deep learning. Theoretical results are complemented by experiments showing more stable, typically lower privacy budgets and faster convergence compared to traditional DP approaches, reducing overestimation in privacy accounting. The framework promises practical impact for private machine learning by enabling tighter, interpretable privacy guarantees without sacrificing utility when large data volumes are available.

Abstract

Differential privacy (DP) has achieved remarkable results in the field of privacy-preserving machine learning. However, existing DP frameworks do not satisfy all the conditions for becoming metrics, which prevents them from deriving better basic private properties and leads to exaggerated values on privacy budgets. We propose Wasserstein differential privacy (WDP), an alternative DP framework to measure the risk of privacy leakage, which satisfies the properties of symmetry and triangle inequality. We show and prove that WDP has 13 excellent properties, which can be theoretical supports for the better performance of WDP than other DP frameworks. In addition, we derive a general privacy accounting method called Wasserstein accountant, which enables WDP to be applied in stochastic gradient descent (SGD) scenarios containing sub-sampling. Experiments on basic mechanisms, compositions and deep learning show that the privacy budgets obtained by Wasserstein accountant are relatively stable and less influenced by order. Moreover, the overestimation on privacy budgets can be effectively alleviated. The code is available at https://github.com/Hifipsysta/WDP.

Figures (4)

• Figure 1: Privacy buget curves of $(\mu, \varepsilon)$-WDP and $(\alpha, \varepsilon)$-RDP for Laplace mechanism (LM) and Gaussian mechanism (GM) with varying orders. Where $\lambda$ and $\sigma$ is the scale of LM and GM respectively. The sensitivities are set to 1 and remains unchanged.
• Figure 2: Privacy budgets over synthetic gradients obtained by moments accountant under DP, Bayesian accountant under BDP and Wasserstein accountant under WDP without gradient clipping.
• Figure 3: The impact of $\beta$ and $\delta$. The coordinates of horizontal axis in \ref{['fig:wdp_delta_epsilon']} are on a logarithmic scale.
• Figure 4: Privacy budgets over synthetic gradients obtained by moments accountant under DP, Bayesian accountant under BDP and Wasserstein accountant under WDP when applying gradient clipping.