Your Privacy Depends on Others: Collusion Vulnerabilities in Individual Differential Privacy

TL;DR

This work reveals a previously overlooked vulnerability in sampling-based iDP mechanisms: while conforming to the iDP guarantees, an individual's privacy risk is not solely governed by their own privacy budget, but critically depends on the privacy choices of all other data contributors.

Abstract

Individual Differential Privacy (iDP) promises users control over their privacy, but this promise can be broken in practice. We reveal a previously overlooked vulnerability in sampling-based iDP mechanisms: while conforming to the iDP guarantees, an individual's privacy risk is not solely governed by their own privacy budget, but critically depends on the privacy choices of all other data contributors. This creates a mismatch between the promise of individual privacy control and the reality of a system where risk is collectively determined. We demonstrate empirically that certain distributions of privacy preferences can unintentionally inflate the privacy risk of individuals, even when their formal guarantees are met. Moreover, this excess risk provides an exploitable attack vector. A central adversary or a set of colluding adversaries can deliberately choose privacy budgets to amplify vulnerabilities of targeted individuals. Most importantly, this attack operates entirely within the guarantees of DP, hiding this excess vulnerability. Our empirical evaluation demonstrates successful attacks against 62% of targeted individuals, substantially increasing their membership inference susceptibility. To mitigate this, we propose $(\varepsilon_i,δ_i,\overlineΔ)$-iDP a privacy contract that uses $Δ$-divergences to provide users with a hard upper bound on their excess vulnerability, while offering flexibility to mechanism design. Our findings expose a fundamental challenge to the current paradigm, demanding a re-evaluation of how iDP systems are designed, audited, communicated, and deployed to make excess risks transparent and controllable.

Figures (10)

• Figure 1: Privacy profiles (left) and adversarial-advantage trade-off curves (right) for subsampled Gaussian mechanisms with varying sampling rates and noise multipliers. While all are calibrated to $(2, 0.08)$-DP, their privacy profiles and thus the adversarial advantage differ substantially, leading to different levels of protection and distinct vulnerabilities.
• Figure 2: Privacy profiles, trade-off functions, and adversarial advantage for a machine learning model trained with sampling-based iDP. While all mechanisms are calibrated to guarantee the predefined privacy budgets $\varepsilon_1 = 8, \varepsilon_2 = 32$ at $\delta=10^{-12}$, their descriptive profiles and consequently the adversarial advantage differ substantially. We obtain the advantage analytically by using the privacy profiles and trade-off functions of the subsampled Gaussian mechanism, parametrized by the sampling rate and noise multiplier. This value serves as a theoretical upper bound on the advantage that can be observed empirically.
• Figure 3: Adversarial advantage on a machine learning model on data points with an $\varepsilon_1 = 8, \delta=10^{-12}$ in the presence of $p$ proportion of data points of $\varepsilon_2$ when privacy is protected via sampling-based iDP. The theoretical MIA advantage and consequently the excess risk strongly depend on the proportion and privacy budget of group 2. If $\varepsilon_2 < \varepsilon_1$, the adversarial risk increases with decreasing $\varepsilon_2$ and increasing proportion of group 2 in the dataset. If $\varepsilon_2 > \varepsilon_1$, the adversarial risk decreases with decreasing $\varepsilon_2$ and also decreases the proportion of group 2 in the dataset.
• Figure 4: Schematisation of the Budget Manipulation Attack. A central adversary (model owner) performs a computation expending at most $\varepsilon_i, \delta$ privacy budget for data point $i$. The adversary deliberately assigns sample-level privacy budgets to increase the excess risk of a targeted data point. Most importantly, the choices of privacy budget are bounded by the stipulated $\varepsilon_i, \delta$ privacy budgets.
• Figure 5: Concept visualization of the collusion attack. Individual data contributors collude on a chosen target (T=1) and choose privacy budgets ($\bar{\varepsilon}_2, \bar{\varepsilon}_3, \dots \bar{\varepsilon}_n$) to increase the vulnerability of the target data.

Theorems & Definitions (8)

• Definition 1: Approximate DP, $(\varepsilon,\delta)$-DP, dwork2006calibrating
• Definition 2: Gaussian Mechanism for $(\varepsilon,\delta)$-DP
• Definition 3: Approximate Individual DP, heo2023personalized
• Definition 4: Sampling with pre-defined expected batch size, boenisch2023have
• Definition 5: Threat Model: Budget Manipulation Attack
• Definition 6: Threat Model: Adversarial Collusion
• Definition 7: $\Delta$-Divergence kaissis2024beyond
• Definition 8: $(\varepsilon_i,\delta_i,\overline{\Delta})$-iDP