A Novel Review of Stability Techniques for Improved Privacy-Preserving Machine Learning

TL;DR

This work tackles privacy leakage in machine learning by leveraging stability as a key lever to reduce the amount of noise needed under PAC privacy. It systematically evaluates stability-enhancing techniques—data transforms from pretrained models, whole-batch gradient clipping, regularization, and group-sample gradient clipping—on CIFAR-10 using Resnet20 and linear regression, including combinations of methods. The findings show that pretrained transforms are highly effective when public data is available, while gradient clipping and regularization offer robust gains in other scenarios; an eigenvalue-based approach enables anisotropic noise to preserve privacy with smaller accuracy loss. The results demonstrate meaningful improvements in stability and provide actionable guidance for privacy-preserving ML, with practical implications for deploying private models that retain higher utility. The study also highlights tradeoffs between stability, accuracy, and privacy guarantees, suggesting directions for future work in data augmentation and dynamic privacy budgeting.

Abstract

Machine learning models have recently enjoyed a significant increase in size and popularity. However, this growth has created concerns about dataset privacy. To counteract data leakage, various privacy frameworks guarantee that the output of machine learning models does not compromise their training data. However, this privatization comes at a cost by adding random noise to the training process, which reduces model performance. By making models more resistant to small changes in input and thus more stable, the necessary amount of noise can be decreased while still protecting privacy. This paper investigates various techniques to enhance stability, thereby minimizing the negative effects of privatization in machine learning.

Figures (10)

• Figure 1: Some samples from the CIFAR-10 dataset
• Figure 2: A tree-shaped model, composed of several binary classifiers, designed for CIFAR-10 DecisionTrees.
• Figure 3: Percentage l2 norms over training time (in epochs), with full batch pictured above and small batch picture below
• Figure 4: Absolute deviation for points removed from a 10,000-point dataset. The number of points removed is on the X axis, and deviation is on the Y axis
• Figure 5: Deviation l2 norm of the CIFAR-100 and Imagenet transforms tracked across 75 epochs for 1,000, 5,000, 10,000, and 20,000 randomly subsampled datapoints. 1,000 points in the top left, 5,000 in the top right, 10,000 in the bottom left, and 20,000 in the bottom right