Sliced Rényi Pufferfish Privacy: Directional Additive Noise Mechanism and Private Learning with Gradient Clipping

TL;DR

This work introduces SRPP, a directionally refined form of Rényi Pufferfish privacy that replaces high-dimensional optimal transport with per-direction 1-D projections, enabling tractable geometry-aware privacy guarantees. It develops sliced Wasserstein mechanisms and SRPP envelopes (SRPE) to calibrate noise for both static queries and iterative learning, avoiding contraction or high-dimensional OT assumptions. For learning, the authors propose SRPP-SGD and ms-SRPP-SGD with History-Uniform Cap (HUC) and mean-square HUC, providing additive composition guarantees across multiple mechanisms. Empirical results across tabular and image datasets show favorable privacy-utility trade-offs, with ms-SRPP requiring substantially less noise than worst-case SRPP and outperforming group-DP baselines in many regimes.

Abstract

We study privatization mechanism design and privacy accounting in the Pufferfish family, addressing two practical gaps of Renyi Pufferfish Privacy (RPP): high-dimensional optimal transport (OT) calibration and the absence of a general, mechanism-agnostic composition rule for iterative learning. We introduce Sliced Renyi Pufferfish Privacy (SRPP), which replaces high-dimensional comparisons by directional ones over a set of unit vectors, enabling geometry-aware and tractable guarantees. To calibrate noise without high-dimensional OT, we propose sliced Wasserstein mechanisms that compute per-direction (1-D) sensitivities, yielding closed-form, statistically stable, and anisotropic calibrations. We further define SRPP Envelope (SRPE) as computable upper bounds that are tightly implementable by these sliced Wasserstein mechanisms. For iterative deep learning algorithms, we develop a decompose-then-compose SRPP-SGD scheme with gradient clipping based on a History-Uniform Cap (HUC), a pathwise bound on one-step directional changes that is uniform over optimization history, and a mean-square variant (ms-HUC) that leverages subsampling randomness to obtain on-average SRPP guarantees with improved utility. The resulting HUC and ms-HUC accountants aggregate per-iteration, per-direction Renyi costs and integrate naturally with moments-accountant style analyses. Finally, when multiple mechanisms are trained and privatized independently under a common slicing geometry, our analysis yields graceful additive composition in both worst-case and mean-square regimes. Our experiments indicate that the proposed SRPP-based methods achieve favorable privacy-utility trade-offs in both static and iterative settings.

Figures (4)

• Figure 1: Geometry and sliced Wasserstein profile for two 2-D Gaussians $\mathrm{P}$ and $\mathrm{Q}$. (a) Geometry of the distributions with selected projection directions: $\mathbf{u}_{\max}$ (purple), $\mathbf{u}_{\min}$ (green), and a typical direction $\mathbf{u}_{\mathrm{typ}}$ (orange), together with the global 2D Wasserstein shift. (b) Polar plot of the normalized 1-D Wasserstein distance as a function of projection angle; the polygon shows per–direction distances, the red dashed circle is their mean, the shaded band shows $\pm 1\mathrm{std}$, and the black dash–dotted circle marks the global $W_2$ distance, with color-matched markers corresponding to the directions in (a).
• Figure 2: Sliced Rényi divergences for $\alpha = 4$ in 2-D, with uniform $\omega$. (a) Two Gaussian distributions $\mathrm{P}$ (red) and $\mathrm{Q}$ (blue). (b)-(e) Polar plots showing divergence profile $\mathtt{D}_{\alpha}$ for $m = 4, 10, 24$ slices and the continuous limit ($m=\infty$). The purple polygon shows per-slice divergences, blue dashed circle shows Ave-SRD (arithmetic mean across directions), and green dotted circle shows Joint-SRD (log-exponential mean emphasizing high-divergence directions). Joint-SRD $\geq$ Ave-SRD in all cases, with both converging as m increases.
• Figure 3: (a)-(d) with $\alpha = 4$ compares Ave-/Joint-SRPE on the balanced Adult dataset (uniform $0.2$ prior over races), for four query families: logistic regression, linear SVM, random forest feature importances, and summary statistics. Each panel plots the utility–privacy tradeoff as a function of the SRPP budget $\epsilon$. (e)-(h) with $\alpha=4$ performs the same tasks on the imbalanced Adult dataset (uniform $0.86$ prior over races). (i) compares SRPP-SGD and ms-SRPP-SGD on CIFAR-10 for two sampling rates ($\eta=0.01$ and $\eta=0.02$), showing the calibrated per–iteration noise scale $\sigma$ (left axis, log–scale) and test accuracy (right axis) as a function of the privacy budget $\epsilon$. (j) shows per-step Gaussian noise scale $\sigma$, and CIFAR-10 test accuracy for group-DP-SGD, SRPP-SGD, and ms-SRPP-SGD under the same clipping, subsampling ($\eta=0.01$ and $\eta=0.02$), and model setup. (k) presents overfitted SRPP-SGD on CIFAR-10 for $\eta = 0.02$. (l) illustrates member/non-member losses and attack AUC of the empirical MIA as a function of the ms-SRPP budget $\epsilon$. All (i)-(l) use $\alpha = 16$. Appendix \ref{['app:experiments']} provides detailed numerical values.
• Figure 4: Privacy-utility tradeoffs on Cleveland Heart Disease and Student Performance datasets. (a)--(d): Cleveland dataset with imbalanced prior (majority-class baseline $\mathrm{Acc}_{\mathrm{maj}}^{0.7} = 0.7$). (e)--(h): Cleveland dataset with balanced prior ($\mathrm{Acc}_{\mathrm{maj}}^{0.5} = 0.5$). (i)--(l): Student Performance dataset with imbalanced prior (sex as secret, $\mathrm{Acc}_{\mathrm{maj}}^{0.6} = 0.6$). (m)--(p): Student Performance dataset with balanced prior (grade group as secret, $\mathrm{Acc}_{\mathrm{maj}}^{0.2} = 0.2$). Each configuration evaluates Ave-SRPP and Joint-SRPP mechanisms across privacy budgets $\epsilon \in \{0.05, 0.1, 0.2, 0.4, 0.8, 1.0, 2.0, 4.0, 8.0, 10.0, 12.0, 14.0, 18.0, 20.0, 25.0, 30.0, 35.0, 40.0, 45.0, 50.0, 55.0, 60.0\}$ with Rényi order $\alpha = 4$. Metrics shown: mean squared error (MSE), attack accuracy, and attack advantage over the priors.

Theorems & Definitions (40)

• Definition 2.1: $(\epsilon, \delta)$-Differential Privacy dwork2006calibrating
• Definition 2.2: Pufferfish privacy (PP) kifer2014pufferfishZhang2022
• Definition 2.3: Rényi Pufferfish privacy (RPP) pierquin2024renyi
• Definition 2.4: Coupling
• Definition 2.5: $\infty$-Wasserstein distance
• Theorem 2.6: General Wasserstein mechanism (GWM) pierquin2024renyi
• Definition 3.1: Sliced Wasserstein Distance
• Definition 4.1: $\omega$-Average Sliced Rényi Divergence ($\omega$-Ave-SRD)
• Definition 4.2: $(\alpha, \epsilon, \omega)$-Ave-SRPP
• Definition 4.3: Joint Sliced Rényi Divergence (Joint-SRD)