Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DP-MicroAdam: Private and Frugal Algorithm for Training and Fine-tuning

Mihaela Hudişteanu, Nikita P. Kalinin, Edwige Cyffers

TL;DR

DP-MicroAdam introduces a memory-efficient, sparsity-aware adaptive optimizer for differential privacy. By integrating top-$k$ gradient selection, error feedback, and quantization with DP noise and clipping, it preserves per-coordinate adaptivity while mitigating DP-induced bias in moment estimates. Theoretical analysis shows an $O(1/\sqrt{T})$ convergence rate up to privacy-dependent constants; empirical results across CIFAR-10, ImageNet, and private transformer fine-tuning demonstrate consistent gains over DP-Adam variants and competitive performance with DP-SGD, with reduced hyperparameter sensitivity and memory requirements. Overall, the approach suggests adaptive optimization can be effectively combined with DP to improve stability and accuracy in private learning.

Abstract

Adaptive optimizers are the de facto standard in non-private training as they often enable faster convergence and improved performance. In contrast, differentially private (DP) training is still predominantly performed with DP-SGD, typically requiring extensive compute and hyperparameter tuning. We propose DP-MicroAdam, a memory-efficient and sparsity-aware adaptive DP optimizer. We prove that DP-MicroAdam converges in stochastic non-convex optimization at the optimal $\mathcal{O}(1/\sqrt{T})$ rate, up to privacy-dependent constants. Empirically, DP-MicroAdam outperforms existing adaptive DP optimizers and achieves competitive or superior accuracy compared to DP-SGD across a range of benchmarks, including CIFAR-10, large-scale ImageNet training, and private fine-tuning of pretrained transformers. These results demonstrate that adaptive optimization can improve both performance and stability under differential privacy.

DP-MicroAdam: Private and Frugal Algorithm for Training and Fine-tuning

TL;DR

DP-MicroAdam introduces a memory-efficient, sparsity-aware adaptive optimizer for differential privacy. By integrating top- gradient selection, error feedback, and quantization with DP noise and clipping, it preserves per-coordinate adaptivity while mitigating DP-induced bias in moment estimates. Theoretical analysis shows an convergence rate up to privacy-dependent constants; empirical results across CIFAR-10, ImageNet, and private transformer fine-tuning demonstrate consistent gains over DP-Adam variants and competitive performance with DP-SGD, with reduced hyperparameter sensitivity and memory requirements. Overall, the approach suggests adaptive optimization can be effectively combined with DP to improve stability and accuracy in private learning.

Abstract

Adaptive optimizers are the de facto standard in non-private training as they often enable faster convergence and improved performance. In contrast, differentially private (DP) training is still predominantly performed with DP-SGD, typically requiring extensive compute and hyperparameter tuning. We propose DP-MicroAdam, a memory-efficient and sparsity-aware adaptive DP optimizer. We prove that DP-MicroAdam converges in stochastic non-convex optimization at the optimal rate, up to privacy-dependent constants. Empirically, DP-MicroAdam outperforms existing adaptive DP optimizers and achieves competitive or superior accuracy compared to DP-SGD across a range of benchmarks, including CIFAR-10, large-scale ImageNet training, and private fine-tuning of pretrained transformers. These results demonstrate that adaptive optimization can improve both performance and stability under differential privacy.

Paper Structure

This paper contains 31 sections, 7 theorems, 59 equations, 1 figure, 10 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Adam in the Non-Private Setting
  4. Challenges for Differentially Private Adam and de-biasing methods
  5. MicroAdam (Non-Private)
  6. The DP-MicroAdam Algorithm
  7. Convergence Guarantees for DP-MicroAdam
  8. Experiments
  9. CIFAR-10
  10. ImageNet
  11. Fine-tuning on CIFAR-10 and CIFAR-100
  12. Conclusion
  13. Theoretical convergence result
  14. Detailed Assumptions
  15. Convergence proof
  16. ...and 16 more sections

Key Result

Theorem 1

Under Assumptions assum:main, for $q_{\omega} \mathrel{\mathop:}= (1+\omega)q < 1$ and a step-size $\eta = \min\{\frac{\epsilon_s}{4LC_0}, \frac{1}{\sqrt{T}}\}$, DP-MicroAdam (Algorithm alg:dp-microadam) with noise parameter $\sigma$ satisfies where $C_0\mathrel{\mathop:}= \sqrt{\frac{\textcolor{myblue}{2 \cdot}4(1+q_{\omega}^2)^3}{(1-q_{\omega}^2)^2}\textcolor{myblue}{(}G \textcolor{myblue}{+\sq

Figures (1)

  • Figure 1: Validation loss over training for CIFAR-10 under $(\varepsilon,\delta)=(8,10^{-5})$. Each subplot corresponds to a different Gaussian noise multiplier $\sigma$.

Theorems & Definitions (14)

  • Theorem 1
  • Lemma 1
  • proof
  • Lemma 2
  • proof
  • Lemma 3
  • proof
  • Lemma 4
  • proof
  • Lemma 5
  • ...and 4 more