On the Privacy Effect of Data Enhancement via the Lens of Memorization

TL;DR

This paper reframes privacy in data-enhanced machine learning through memorization, arguing that traditional MIAs can misrepresent per-sample risk. By employing LiRA as a memorization-aligned attack, it demonstrates a weaker link between privacy leakage and generalization gap and that adversarial training increases memorization-based leakage, while stronger robustness does not inherently worsen privacy. The study provides extensive experiments across CIFAR datasets and SVHN, highlighting that data augmentation is not a universal defense against MIAs and that memorization-aligned MIAs are necessary for accurate privacy assessment. The findings advocate for a shift toward memorization-based privacy evaluation in ML systems and reveal nuanced interactions between privacy, generalization, and robustness with practical implications for secure model design.

Abstract

Machine learning poses severe privacy concerns as it has been shown that the learned models can reveal sensitive information about their training data. Many works have investigated the effect of widely adopted data augmentation and adversarial training techniques, termed data enhancement in the paper, on the privacy leakage of machine learning models. Such privacy effects are often measured by membership inference attacks (MIAs), which aim to identify whether a particular example belongs to the training set or not. We propose to investigate privacy from a new perspective called memorization. Through the lens of memorization, we find that previously deployed MIAs produce misleading results as they are less likely to identify samples with higher privacy risks as members compared to samples with low privacy risks. To solve this problem, we deploy a recent attack that can capture individual samples' memorization degrees for evaluation. Through extensive experiments, we unveil several findings about the connections between three essential properties of machine learning models, including privacy, generalization gap, and adversarial robustness. We demonstrate that the generalization gap and privacy leakage are less correlated than those of the previous results. Moreover, there is not necessarily a trade-off between adversarial robustness and privacy as stronger adversarial robustness does not make the model more susceptible to privacy attacks.

Figures (12)

• Figure 1: Feature score (top) and TPR (bottom) of each bin in terms of memorization score for a target model trained on CIFAR-100 using (a) MaxPreCA salem2018ml, (b) Loss attack yeom2018privacy, (c) Modified entropy attack song2021systematic, (d) Binary classifier shokri2017membership, (e) Bayes calibrated loss sablayrolles2019white, (f) Difficulty calibrated loss importance, and (g) LiRA carlini2021membership.
• Figure 2: Memorization scores of 5,000 randomly selected samples using Jitter (top), Disturblabel (middle) and PGD-AT model (bottom) v.s., Base model.
• Figure 3: Attack success rate versus the train-test gap of different data augmentation models on CIFAR-10, CIFAR-100, and SVHN using MaxPreCA (top) and LiRA (bottom), respectively. $r$ stands for the Pearson correlation coefficient.
• Figure 4: Attack success rates of a single query and multiple queries in two cases: augmentation-unaware (left) and augmentation-aware (right). We evaluated different data augmentation methods on CIFAR-10 and CIFAR-100 datasets, respectively. None stands for models trained without any data augmentation.
• Figure 5: The distributions of normalized confidence $\phi$ of three samples with different memorization scores using Base and four adversarial training models. Each row corresponds to a sample.