Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Data Efficient Subset Training with Differential Privacy

Ninad Jayesh Gandhi, Moparthy Venkata Subrahmanya Sri Harsha

TL;DR

The paper tackles data-efficient training under differential privacy by adapting GLISTER to the private setting, creating GLISTER-DP that combines DP-SGD for inner training and a DP submodular exponential-mechanism for subset selection. It formalizes a two-level objective with a subset $S$ of size $|S|\le k$ and demonstrates how the privacy budget is split into $\varepsilon_g$ for training and $\varepsilon_{ss}$ for subset selection. Empirically, full training (FULL-DP) remains strongest, while RANDOM-DP often outperforms GLISTER-DP for $\varepsilon \in \{3,8\}$, indicating that the privacy budget allocated to subset search is too restrictive to yield high-quality subsets. The results imply practical private data-efficient training remains challenging, with the main implication being to allocate more budget to training rather than subset selection and to explore improved DP-based subset selection methods.

Abstract

Private machine learning introduces a trade-off between the privacy budget and training performance. Training convergence is substantially slower and extensive hyper parameter tuning is required. Consequently, efficient methods to conduct private training of models is thoroughly investigated in the literature. To this end, we investigate the strength of the data efficient model training methods in the private training setting. We adapt GLISTER (Killamsetty et al., 2021b) to the private setting and extensively assess its performance. We empirically find that practical choices of privacy budgets are too restrictive for data efficient training in the private setting.

Data Efficient Subset Training with Differential Privacy

TL;DR

The paper tackles data-efficient training under differential privacy by adapting GLISTER to the private setting, creating GLISTER-DP that combines DP-SGD for inner training and a DP submodular exponential-mechanism for subset selection. It formalizes a two-level objective with a subset of size and demonstrates how the privacy budget is split into for training and for subset selection. Empirically, full training (FULL-DP) remains strongest, while RANDOM-DP often outperforms GLISTER-DP for , indicating that the privacy budget allocated to subset search is too restrictive to yield high-quality subsets. The results imply practical private data-efficient training remains challenging, with the main implication being to allocate more budget to training rather than subset selection and to explore improved DP-based subset selection methods.

Abstract

Private machine learning introduces a trade-off between the privacy budget and training performance. Training convergence is substantially slower and extensive hyper parameter tuning is required. Consequently, efficient methods to conduct private training of models is thoroughly investigated in the literature. To this end, we investigate the strength of the data efficient model training methods in the private training setting. We adapt GLISTER (Killamsetty et al., 2021b) to the private setting and extensively assess its performance. We empirically find that practical choices of privacy budgets are too restrictive for data efficient training in the private setting.

Paper Structure

This paper contains 12 sections, 2 equations, 6 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Problem Formulation and Methodology
  4. Problem Formulation
  5. Differentially Private Data Efficient Training
  6. Experiments and Discussion
  7. Conclusion
  8. Experimental Details
  9. GLISTER vs GLISTER-DP
  10. Class Imbalanced Synthetic Datasets
  11. Timing Analysis
  12. Experiments with Allocation Rate.

Figures (6)

  • Figure 1: Performance of GLISTER-DP, RANDOM-DP and FULL-DP on test set for CIFAR10 and MNIST with $\varepsilon \in \{3,8\}$ across various choices of subset size $k$ as a fraction of $|\mathcal{D}|$
  • Figure 2: Comparison of original distribution with the one that exponential mechanism samples from. The plot is generated by resampling the true gains and noisy gains and normalizing to produce a valid probability distribution.
  • Figure 3: Training convergence for each method on CIFAR10, $\varepsilon=3$ and $k=0.5|\mathcal{D}|$
  • Figure 4: Performance comparison on highly imbalanced synthetic dataset.
  • Figure 5: Training convergence plot of GLISTER-DP, FULL-DP and RANDOM-DP across different fractions of training budget on CIFAR-10 $\epsilon=3$
  • ...and 1 more figures

Theorems & Definitions (1)

  • Definition 1.1: $(\varepsilon, \delta)$- Differential Privacy