Beyond the Calibration Point: Mechanism Comparison in Differential Privacy

TL;DR

The paper addresses the inadequacy of evaluating DP mechanisms with a single $(\varepsilon, \delta)$ pair by introducing the $\Delta$-divergence, which quantifies the worst-case excess privacy vulnerability between mechanisms across $\varepsilon, \delta$-DP and $f$-DP, with a Bayesian interpretation. It develops a principled, approximate Blackwell ordering, connects it to Bayes errors via $R_{\min}(\pi)$, and proves that the space of DP mechanisms forms a metric lattice under the symmetrised $\Delta$-divergence $\Delta^{\leftrightarrow}$. The work further shows that under composition, non-dominated mechanisms can become ordered (emergent Blackwell dominance), with explicit finite-sample bounds derived from Gaussian convergence of trade-off functions. Through experiments on DP-SGD and canonical noise-adding mechanisms, the authors demonstrate that calibrating to a single $(\varepsilon, \delta)$ can obscure meaningful privacy-vulnerability differences and that $\Delta$ provides a fine-grained, decision-relevant risk measure. Overall, the framework enables principled, granular mechanism selection and auditing for privacy-preserving ML, highlighting practical risks and guiding more robust DP deployments.

Abstract

In differentially private (DP) machine learning, the privacy guarantees of DP mechanisms are often reported and compared on the basis of a single $(\varepsilon, δ)$-pair. This practice overlooks that DP guarantees can vary substantially even between mechanisms sharing a given $(\varepsilon, δ)$, and potentially introduces privacy vulnerabilities which can remain undetected. This motivates the need for robust, rigorous methods for comparing DP guarantees in such cases. Here, we introduce the $Δ$-divergence between mechanisms which quantifies the worst-case excess privacy vulnerability of choosing one mechanism over another in terms of $(\varepsilon, δ)$, $f$-DP and in terms of a newly presented Bayesian interpretation. Moreover, as a generalisation of the Blackwell theorem, it is endowed with strong decision-theoretic foundations. Through application examples, we show that our techniques can facilitate informed decision-making and reveal gaps in the current understanding of privacy risks, as current practices in DP-SGD often result in choosing mechanisms with high excess privacy vulnerabilities.

Figures (8)

• Figure 1: Privacy profiles of two SGMs with different noise scales $\sigma$ and sampling rates $p$. Both satisfy $(4.00, 0.08)$-DP, but offer otherwise different levels of privacy protection.
• Figure 2: The trade-off functions of a Gaussian ($f$, blue) and a Laplace ($\widetilde{f}$, red) mechanism cross, therefore neither mechanism universally Blackwell dominates. Shifting $f$ left and down by $\kappa$ yields $f(\alpha) \leq \widetilde{f}(\alpha)$ for all $\alpha \in [0,1]$ (dashed blue), and restores universal Blackwell dominance.
• Figure 3: Approximate comparison between the Laplace ($b=1$) and the Gaussian ($\sigma=1$) mechanism via their Bayes error representations. Note that all divergence values of interest are visually depicted in this representation.
• Figure 4: Hierarchical Bayesian modelling of adversarial strategies. Hyper-prior densities are plotted in the insets.
• Figure 5: Validation accuracy (red) and $\Delta$-divergence values (blue) for mechanisms satisfying $(8, 10^{-5})$-DP. The combinations to the right of $\star$ are not Pareto efficient in terms of balancing accuracy and excess vulnerability over $\mathcal{M}$.

Theorems & Definitions (43)

• theorem 1
• definition 1
• definition 2
• theorem 2
• corollary 1
• definition 3: Symmetrised $\Delta$-divergence $\Delta^{\leftrightarrow}$
• lemma 1
• corollary 2
• lemma 2
• lemma 3