Curator Attack: When Blackbox Differential Privacy Auditing Loses Its Power

TL;DR

This work identified that blackbox auditing is essentially flawed with its setting --- small probabilities/densities are ignored due to inaccurate observation, and reveals the limitations of blackbox auditing tools, empower the data owner with the awareness of risks in using these tools, and encourage the development of more reliable differential privacy auditing methods.

Abstract

A surge in data-driven applications enhances everyday life but also raises serious concerns about private information leakage. Hence many privacy auditing tools are emerging for checking if the data sanitization performed meets the privacy standard of the data owner. Blackbox auditing for differential privacy is particularly gaining popularity for its effectiveness and applicability to a wide range of scenarios. Yet, we identified that blackbox auditing is essentially flawed with its setting: small probabilities or densities are ignored due to inaccurate observation. Our argument is based on a solid false positive analysis from a hypothesis testing perspective, which is missed out by prior blackbox auditing tools. This oversight greatly reduces the reliability of these tools, as it allows malicious or incapable data curators to pass the auditing with an overstated privacy guarantee, posing significant risks to data owners. We demonstrate the practical existence of such threats in classical differential privacy mechanisms against four representative blackbox auditors with experimental validations. Our findings aim to reveal the limitations of blackbox auditing tools, empower the data owner with the awareness of risks in using these tools, and encourage the development of more reliable differential privacy auditing methods.

Figures (12)

• Figure 1: Illustration of the mechanism's strongest achievable DP guarantee $\epsilon^*$ and the its privacy claim $\epsilon_c$.
• Figure 2: (a) An illustration of the FP, FN, TP, and TN regions divided by the true privacy level $\epsilon^{*}$, the claimed level $\epsilon_c$, and maximal power $\xi^*$. (b)(c) DP vs. DP against blackbox audit. FPs exist since the audit ignores small probabilities (densities). (d) Delta-Siege uses a privacy surrogate $\rho$ to seek the optimal power $\xi^*$. Different $\rho$s lead to FPs or FNs depending on the position with the theoretical $\epsilon-\delta$ DP curve. The illustrative privacy surrogates are $\rho_1(\epsilon,\delta)=e^{-3\epsilon}/\delta$ and $\rho_2(\epsilon,\delta)=e^{-2\epsilon}/\delta$.
• Figure 3: DP-Sniper's $\hat{S}$ against the benchmark Laplace $M^{\text{lap}}_{\theta}$. The theoretical optimal set is $S^*=(-\infty,0]$.
• Figure 6: The benchmark Laplace mechanism is an FP against DP-Sniper at $\theta>-\ln(2c)$.
• Figure 7: Adapted Laplace mechanism against DP-Sniper's auditing, $c=0.01$ or $0.05$. All adapted mechanisms are FPs.

Theorems & Definitions (12)

• Definition 1: True Privacy Level $\epsilon^*$
• Definition 2: Maximal Power $\xi^*$
• Definition 3: False Positives
• Theorem 1
• Theorem 2
• Theorem 3
• Theorem 4
• Theorem 5
• Theorem 6
• Theorem 7