The Tail Tells All: Estimating Model-Level Membership Inference Vulnerability Without Reference Models

TL;DR

This work tackles the practicality gap in evaluating membership inference risk by proposing a reference-free metric for model-level vulnerability. It demonstrates that loss distributions are heavy-tailed and that vulnerable samples are those missing from the tail, enabling the use of the LOSS TNR (and LOSS AUC) as predictors of LiRA risk without training reference models. Nonlinear fits, notably an exponential mapping, offer accurate estimates of LiRA TPR@FPR=$10^{-3}$ across diverse architectures and four image datasets, with strong results even for LLMs. The approach outperforms low-cost baselines like RMIA and baseline distribution differences, offering a scalable tool for privacy risk assessment during iterative model development.

Abstract

Membership inference attacks (MIAs) have emerged as the standard tool for evaluating the privacy risks of AI models. However, state-of-the-art attacks require training numerous, often computationally expensive, reference models, limiting their practicality. We present a novel approach for estimating model-level vulnerability, the TPR at low FPR, to membership inference attacks without requiring reference models. Empirical analysis shows loss distributions to be asymmetric and heavy-tailed and suggests that most points at risk from MIAs have moved from the tail (high-loss region) to the head (low-loss region) of the distribution after training. We leverage this insight to propose a method to estimate model-level vulnerability from the training and testing distribution alone: using the absence of outliers from the high-loss region as a predictor of the risk. We evaluate our method, the TNR of a simple loss attack, across a wide range of architectures and datasets and show it to accurately estimate model-level vulnerability to the SOTA MIA attack (LiRA). We also show our method to outperform both low-cost (few reference models) attacks such as RMIA and other measures of distribution difference. We finally evaluate the use of non-linear functions to evaluate risk and show the approach to be promising to evaluate the risk in large-language models.

Figures (8)

• Figure 1: Histograms showing loss distributions for training set members (orange) and non-members (blue) in log-log scale across three neural network architectures trained on CINIC-10: DenseNet121, WRN40-4, and ResNet-18. The distributions demonstrate clear separation between members and non-members, with members typically having lower loss values.
• Figure 2: Loss distributions for members and non-members for ResNet-18 (CINIC10). Orange histogram shows member losses, blue shows non-member losses, with density curves overlaid. Green bars indicate the OUT model mean ($l_{out}$) for samples identified by LiRA at FPR=0.001.
• Figure 3: The LiRA TPR@FPR=0.001 as a linear function of the TNR@FPR for across all setups with the 97.5% confidence interval.
• Figure 4: Performance of linear models for predicting LiRA with varying numbers of reference models (K). Left panel shows the correlation between LOSS TNR (x-axis) and LiRA TPR@0.001 (y-axis) across different K values, with linear regression fits shown for each condition. Right panel shows slope parameter ($\alpha$) for different K. The error bars show the 97.5% confidence intervals obtained with boostrapping.
• Figure 5: The distributions of members and non-members (left). The LIRA TPR@FPR LOSS as a linear function of the LOSS AUC evaluated on LLMs.