Hyperparameters in Score-Based Membership Inference Attacks

TL;DR

The paper investigates privacy leakage in ML through score-based membership inference attacks (MIAs) and questions whether an attacker must know the target model's hyperparameters in transfer learning. It introduces KL-LiRA, a hyperparameter-selection method that minimizes the KL divergence between Gaussian approximations of the target and shadow loss distributions, phi_KL(N_T||N_S) = 1/2[ (mu_S - mu_T)^2 / sigma_S^2 + sigma_T^2 / sigma_S^2 - ln(sigma_T^2 / sigma_S^2) - 1 ]. Empirically KL-LiRA matches LiRA that uses target hyperparameters in non-DP settings, while ACC-LiRA underperforms; DP reduces attack effectiveness overall. Moreover, examining hyperparameter optimization under differential privacy shows no statistically significant increase in MIA vulnerability when HPO uses training data versus external data, implying limited privacy risk from DP-HPO; the work informs practical HPO strategies and privacy assessments for DP transfer learning.

Abstract

Membership Inference Attacks (MIAs) have emerged as a valuable framework for evaluating privacy leakage by machine learning models. Score-based MIAs are distinguished, in particular, by their ability to exploit the confidence scores that the model generates for particular inputs. Existing score-based MIAs implicitly assume that the adversary has access to the target model's hyperparameters, which can be used to train the shadow models for the attack. In this work, we demonstrate that the knowledge of target hyperparameters is not a prerequisite for MIA in the transfer learning setting. Based on this, we propose a novel approach to select the hyperparameters for training the shadow models for MIA when the attacker has no prior knowledge about them by matching the output distributions of target and shadow models. We demonstrate that using the new approach yields hyperparameters that lead to an attack near indistinguishable in performance from an attack that uses target hyperparameters to train the shadow models. Furthermore, we study the empirical privacy risk of unaccounted use of training data for hyperparameter optimization (HPO) in differentially private (DP) transfer learning. We find no statistically significant evidence that performing HPO using training data would increase vulnerability to MIA.

Figures (1)

• Figure 1: Comparing the performance of attacks using different hyperparameter selection strategies. Selecting the hyperparameters that maximize the similarity between the output distribution of scores of the target model and the shadow model (LiRA using distribution matching) yields an attack as powerful as LiRA when target hyperparameters are used to train the shadow models. On the other hand, training shadow models with the hyperparameters that yield maximum test accuracy on the shadow data sets (LiRA using optimal test accuracy) yields a significantly weaker attack. This shows that selecting hyperparameters for training shadow models is a non-trivial aspect of MIAs.