Low-Cost High-Power Membership Inference Attacks

TL;DR

This paper tackles the practical privacy auditing problem of membership inference attacks by introducing RMIA, a low-cost, high-power test that leverages a fine-grained null model and a Bayesian-composed likelihood-ratio framework. RMIA combines information from reference population data and pre-trained reference models to form pairwise likelihood ratios, aggregating them into a single MIA score that can calibrate to any desired false-positive rate. Empirical results across CIFAR-10/100, CINIC-10, ImageNet, and Purchase-100 show RMIA consistently dominates prior attacks, especially under limited reference models and under distribution and architecture shifts, with strong robustness to OOD non-members. The approach enables practical privacy-risk assessment and auditing, reducing computational overhead while maintaining strong leakage detection, even for large-scale models and varied data distributions.

Abstract

Membership inference attacks aim to detect if a particular data point was used in training a model. We design a novel statistical test to perform robust membership inference attacks (RMIA) with low computational overhead. We achieve this by a fine-grained modeling of the null hypothesis in our likelihood ratio tests, and effectively leveraging both reference models and reference population data samples. RMIA has superior test power compared with prior methods, throughout the TPR-FPR curve (even at extremely low FPR, as low as 0). Under computational constraints, where only a limited number of pre-trained reference models (as few as 1) are available, and also when we vary other elements of the attack (e.g., data distribution), our method performs exceptionally well, unlike prior attacks that approach random guessing. RMIA lays the groundwork for practical yet accurate data privacy risk assessment in machine learning.

Figures (26)

• Figure 1: RMIA versus the prior attacks, Attack-P and Attack-R Ye2022Enhanced and also LiRA Carlini2022Membership, on CIFAR-100 models, with the restriction of using only $1$ reference model (in an offline setting). RMIA outperforms other attacks throughout the TPR-FPR trade-off curve (e.g. by at least 25% higher AUC and an order of magnitude better TPR at zero FPR, compared with LiRA).
• Figure 2: ROC of attacks against ImageNet models. The result is obtained on one random target model. We use 1 reference model (OUT). Table \ref{['tab:cifar_cinic_imagenet_models_1_2_4']} reports AUC of attacks.
• Figure 3: Number of reference models versus the AUC of the attacks on CIFAR-10. In online attacks, half of reference models need to be trained per each MIA query.
• Figure 4: ROC of offline attacks using models trained on CIFAR-10, while non-member test queries are OOD samples from CINIC-10. We use 127 reference models.
• Figure 5: AUC of our attack (RMIA) obtained by using different values of $n$ (order in Taylor function), $m$ (soft-margin) and $T$ (temperature) in SM-Taylor-Softmax function. When modifying one parameter, we hold the values of the other two parameters constant at their optimal values. Here, we use 254 reference models trained on CIFAR-10. Results are averaged over 10 target models. The red points indicate the default values used in our experiments.

Theorems & Definitions (2)

• Definition 2.1: Membership Inference Game
• Definition 3.1: Robust Membership Inference Attack