Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Revisiting Membership Inference Under Realistic Assumptions

Bargav Jayaraman, Lingxiao Wang, Katherine Knipmeyer, Quanquan Gu, David Evans

TL;DR

This work revisits membership inference under realistic conditions by allowing imbalanced priors and threshold-adaptive adversaries. It introduces a PPV-based leakage metric and two novel attacks, Merlin and Morgan, alongside a threshold-selection procedure to tailor attacks to defender goals. Grounded in f-DP and Gaussian DP theory, the paper provides theoretical bounds and extensive empirical evaluation across four multi-class datasets, with and without differential privacy. The results show persistent privacy risks for non-private models under imbalanced priors, and that DP reduces leakage only at strong budgets, underscoring the need for prior-aware evaluation and threshold-aware defenses.

Abstract

We study membership inference in settings where some of the assumptions typically used in previous research are relaxed. First, we consider skewed priors, to cover cases such as when only a small fraction of the candidate pool targeted by the adversary are actually members and develop a PPV-based metric suitable for this setting. This setting is more realistic than the balanced prior setting typically considered by researchers. Second, we consider adversaries that select inference thresholds according to their attack goals and develop a threshold selection procedure that improves inference attacks. Since previous inference attacks fail in imbalanced prior setting, we develop a new inference attack based on the intuition that inputs corresponding to training set members will be near a local minimum in the loss function, and show that an attack that combines this with thresholds on the per-instance loss can achieve high PPV even in settings where other attacks appear to be ineffective. Code for our experiments can be found here: https://github.com/bargavj/EvaluatingDPML.

Revisiting Membership Inference Under Realistic Assumptions

TL;DR

This work revisits membership inference under realistic conditions by allowing imbalanced priors and threshold-adaptive adversaries. It introduces a PPV-based leakage metric and two novel attacks, Merlin and Morgan, alongside a threshold-selection procedure to tailor attacks to defender goals. Grounded in f-DP and Gaussian DP theory, the paper provides theoretical bounds and extensive empirical evaluation across four multi-class datasets, with and without differential privacy. The results show persistent privacy risks for non-private models under imbalanced priors, and that DP reduces leakage only at strong budgets, underscoring the need for prior-aware evaluation and threshold-aware defenses.

Abstract

We study membership inference in settings where some of the assumptions typically used in previous research are relaxed. First, we consider skewed priors, to cover cases such as when only a small fraction of the candidate pool targeted by the adversary are actually members and develop a PPV-based metric suitable for this setting. This setting is more realistic than the balanced prior setting typically considered by researchers. Second, we consider adversaries that select inference thresholds according to their attack goals and develop a threshold selection procedure that improves inference attacks. Since previous inference attacks fail in imbalanced prior setting, we develop a new inference attack based on the intuition that inputs corresponding to training set members will be near a local minimum in the loss function, and show that an attack that combines this with thresholds on the per-instance loss can achieve high PPV even in settings where other attacks appear to be ineffective. Code for our experiments can be found here: https://github.com/bargavj/EvaluatingDPML.

Paper Structure

This paper contains 24 sections, 8 theorems, 12 equations, 13 figures, 10 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Differential Privacy
  4. f-Differential Privacy
  5. Gaussian Differential Privacy
  6. Measuring Privacy Leakage
  7. Membership Advantage
  8. Positive Predictive Value
  9. Inference Attacks
  10. Setting the Decision Threshold
  11. Merlin
  12. Morgan
  13. Experimental Setup
  14. Empirical Results
  15. Yeom Attack
  16. ...and 9 more sections

Key Result

Lemma 3.4

Suppose $\mathcal{M}$ is an $(\epsilon, \delta)$-differentially private algorithm, then for a false positive rate of $\alpha$, the trade-off function is:

Figures (13)

  • Figure 1: Theoretical upper bounds on $\mathit{Adv}_\mathcal{A}(\alpha)$ metric for various privacy loss budgets with varying $\alpha$ ($\delta = 10^{-5}$).
  • Figure 2: Comparing theoretical bounds on membership advantage ($\delta = 0$). Improved bound uses Theorem \ref{['thm:adv']} to get maximum advantage across all $0 < \alpha \le 1$.
  • Figure 3: Theoretical upper bounds on PPV metric for various privacy budgets ($\delta = 10^{-5}$).
  • Figure 4: Accuracy loss comparison of private models trained with different privacy analyses.
  • Figure 5: Analysis of Yeom on non-private model trained on Purchase-100X with balanced prior. The x-axis shows the per-instance loss on a logarithmic scale from $10^{-7}$ to $10^1$ where the buckets are in the range $(10^{-7}, 10^{-6.9})$, $(10^{-6.9}, 10^{-6.8})$, and so on up to $(10^{0.9}, 10^{1})$.
  • ...and 8 more figures

Theorems & Definitions (14)

  • Definition 3.1: Differential Privacy
  • Definition 3.2: Trade-off Function
  • Definition 3.3: $f$-Differential Privacy
  • Lemma 3.4: wasserman2010statisticalkairouz2017composition
  • Definition 3.5: $\mu$-Gaussian Differential Privacy
  • Lemma 3.6
  • Theorem 3.7
  • Proposition 3.8
  • Theorem 3.9: Composition
  • Theorem 3.10: Sub-sampling
  • ...and 4 more