Effectiveness of L2 Regularization in Privacy-Preserving Machine Learning

TL;DR

This paper addresses privacy risks in ML by focusing on Membership Inference Attacks and evaluating whether $L_2$ regularization can mitigate these attacks as effectively as, or better than, Differential Privacy (DP). By analyzing Deep Neural Networks across MNIST, CIFAR-10, and an augmented Toxic Tweets dataset, the study shows that mild $L_2$ regularization reduces attacker advantage and sustains or improves validation accuracy, especially in non-DP settings, while DP provides strong privacy but often at the cost of utility. The results highlight a strong link between overfitting (training–evaluation accuracy gap) and MIAs, with $L_2$ regularization helping to shrink this gap and diminish attack efficacy, sometimes outperforming DP in both privacy and performance. The paper discusses limitations, notably the lack of formal privacy guarantees with $L_2$ regularization and the potential need to combine it with DP or other PPML techniques for robust protection in diverse data regimes.

Abstract

Artificial intelligence, machine learning, and deep learning as a service have become the status quo for many industries, leading to the widespread deployment of models that handle sensitive data. Well-performing models, the industry seeks, usually rely on a large volume of training data. However, the use of such data raises serious privacy concerns due to the potential risks of leaks of highly sensitive information. One prominent threat is the Membership Inference Attack, where adversaries attempt to deduce whether a specific data point was used in a model's training process. An adversary's ability to determine an individual's presence represents a significant privacy threat, especially when related to a group of users sharing sensitive information. Hence, well-designed privacy-preserving machine learning solutions are critically needed in the industry. In this work, we compare the effectiveness of L2 regularization and differential privacy in mitigating Membership Inference Attack risks. Even though regularization techniques like L2 regularization are commonly employed to reduce overfitting, a condition that enhances the effectiveness of Membership Inference Attacks, their impact on mitigating these attacks has not been systematically explored.

Figures (4)

• Figure 1: $L_2$ regularization on the FCNN model trained on the MNIST dataset. The figure shows how validation accuracy and attacker advantage change with different $L_2$ regularization strengths ($\lambda$) for both Baseline (non-DP) and DP models.
• Figure 2: Effect of $L_2$ regularization on the CNN model trained on the CIFAR10 dataset. The figure illustrates the relationship between validation accuracy and attacker advantage across different $L_2$ regularization strengths ($\lambda$) for both Baseline (non-DP) and DP models.
• Figure 3: Effect of $L_2$ regularization on the text classification task. The top plot shows the validation accuracy across different $L_2$ regularization strengths ($\lambda$) for both Baseline (non-DP) and DP models. The bottom plot illustrates the corresponding attacker advantage for each model.
• Figure 4: Correlation between the accuracy difference (training accuracy minus validation accuracy) and attacker advantage. The plot demonstrates a strong positive correlation (correlation coefficient of 0.93).