Statistical MIA: Rethinking Membership Inference Attack for Reliable Unlearning Auditing

TL;DR

This work questions the reliability of MIA-based auditing for machine unlearning and shows that attack failures do not guarantee forgetting. It introduces SMIA, a training-free framework that uses distributional distance metrics to quantify forgetting and provide confidence intervals, thereby avoiding shadow-model overhead. Through three variants—SMIA-0 (moments), SMIA-M (kernel mean embeddings), and SMIA-W (entropy-regularized Wasserstein)—the paper demonstrates that SMIA-M offers robust, efficient auditing that outperforms prior MIA baselines. The approach enables more trustworthy auditing in practice and suggests a new paradigm for reliable unlearning verification with broad applicability in privacy-preserving ML.

Abstract

Machine unlearning (MU) is essential for enforcing the right to be forgotten in machine learning systems. A key challenge of MU is how to reliably audit whether a model has truly forgotten specified training data. Membership Inference Attacks (MIAs) are widely used for unlearning auditing, where samples that evade membership detection are often regarded as successfully forgotten. After carefully revisiting the reliability of MIA, we show that this assumption is flawed: failed membership inference does not imply true forgetting. We theoretically demonstrate that MIA-based auditing, when formulated as a binary classification problem, inevitably incurs statistical errors whose magnitude cannot be observed during the auditing process. This leads to overly optimistic evaluations of unlearning performance, while incurring substantial computational overhead due to shadow model training. To address these limitations, we propose Statistical Membership Inference Attack (SMIA), a novel training-free and highly effective auditing framework. SMIA directly compares the distributions of member and non-member data using statistical tests, eliminating the need for learned attack models. Moreover, SMIA outputs both a forgetting rate and a corresponding confidence interval, enabling quantified reliability of the auditing results. Extensive experiments show that SMIA provides more reliable auditing with significantly lower computational cost than existing MIA-based approaches. Notably, the theoretical guarantees and empirical effectiveness of SMIA suggest it as a new paradigm for reliable machine unlearning auditing.

Figures (6)

• Figure 1: The relationship between the proportion of non-member data and successful TNR detection, under the configuration of MIA accuracy=0.99 and member detection success rate 0.9999.
• Figure 2: (a) The dilemma faced by the attacker; (b) The dilemma faced by the auditer.
• Figure 3: An example of audit failure
• Figure 4: Box plots of the auditing performance of SMIA-0 and SMIA-M under different numbers of bootstrap groups.
• Figure 5: Relationship between the number of audit samples and the auditing performance of SMIA-0 and SMIA-M.

Theorems & Definitions (17)

• Theorem 2.1: MIA Error Decomposition via Empirical Risk
• Corollary 2.2: Auditing Error Decomposition via Empirical Risk
• Remark 3.1
• Proposition 3.2
• Example 3.1: Gradient-based White-box Neuron Auditing Instance
• Lemma 3.3
• Remark 3.4
• Definition 3.5
• Remark 3.1
• Lemma 3.2