Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Understanding the Impact of Differentially Private Training on Memorization of Long-Tailed Data

Jiaming Zhang, Huanyi Xie, Meng Ding, Shaopeng Fu, Jinyan Liu, Di Wang

TL;DR

This work develops a theoretical feature-learning framework to understand how DP-SGD affects memorization on long-tailed data, using a multi-class, two-layer CNN with class-dependent noise. It shows that gradient clipping and privacy noise can suppress memorization of rare but informative patterns, but DP-induced noise exacerbates test errors for long-tailed subpopulations, revealing a pronounced privacy-utility trade-off. The authors provide formal results on training dynamics and two regimes of generalization, and validate them with synthetic experiments and real datasets (MNIST, CIFAR-10), including a practical method to identify long-tailed samples via influence scores. Overall, the paper highlights the challenges of private deep learning on imbalanced data and informs design choices for DP methods that preserve utility for underrepresented groups.

Abstract

Recent research shows that modern deep learning models achieve high predictive accuracy partly by memorizing individual training samples. Such memorization raises serious privacy concerns, motivating the widespread adoption of differentially private training algorithms such as DP-SGD. However, a growing body of empirical work shows that DP-SGD often leads to suboptimal generalization performance, particularly on long-tailed data that contain a large number of rare or atypical samples. Despite these observations, a theoretical understanding of this phenomenon remains largely unexplored, and existing differential privacy analysis are difficult to extend to the nonconvex and nonsmooth neural networks commonly used in practice. In this work, we develop the first theoretical framework for analyzing DP-SGD on long-tailed data from a feature learning perspective. We show that the test error of DP-SGD-trained models on the long-tailed subpopulation is significantly larger than the overall test error over the entire dataset. Our analysis further characterizes the training dynamics of DP-SGD, demonstrating how gradient clipping and noise injection jointly adversely affect the model's ability to memorize informative but underrepresented samples. Finally, we validate our theoretical findings through extensive experiments on both synthetic and real-world datasets.

Understanding the Impact of Differentially Private Training on Memorization of Long-Tailed Data

TL;DR

This work develops a theoretical feature-learning framework to understand how DP-SGD affects memorization on long-tailed data, using a multi-class, two-layer CNN with class-dependent noise. It shows that gradient clipping and privacy noise can suppress memorization of rare but informative patterns, but DP-induced noise exacerbates test errors for long-tailed subpopulations, revealing a pronounced privacy-utility trade-off. The authors provide formal results on training dynamics and two regimes of generalization, and validate them with synthetic experiments and real datasets (MNIST, CIFAR-10), including a practical method to identify long-tailed samples via influence scores. Overall, the paper highlights the challenges of private deep learning on imbalanced data and informs design choices for DP methods that preserve utility for underrepresented groups.

Abstract

Recent research shows that modern deep learning models achieve high predictive accuracy partly by memorizing individual training samples. Such memorization raises serious privacy concerns, motivating the widespread adoption of differentially private training algorithms such as DP-SGD. However, a growing body of empirical work shows that DP-SGD often leads to suboptimal generalization performance, particularly on long-tailed data that contain a large number of rare or atypical samples. Despite these observations, a theoretical understanding of this phenomenon remains largely unexplored, and existing differential privacy analysis are difficult to extend to the nonconvex and nonsmooth neural networks commonly used in practice. In this work, we develop the first theoretical framework for analyzing DP-SGD on long-tailed data from a feature learning perspective. We show that the test error of DP-SGD-trained models on the long-tailed subpopulation is significantly larger than the overall test error over the entire dataset. Our analysis further characterizes the training dynamics of DP-SGD, demonstrating how gradient clipping and noise injection jointly adversely affect the model's ability to memorize informative but underrepresented samples. Finally, we validate our theoretical findings through extensive experiments on both synthetic and real-world datasets.
Paper Structure (20 sections, 28 theorems, 145 equations, 4 figures)

This paper contains 20 sections, 28 theorems, 145 equations, 4 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Problem Setup
  4. Main Results
  5. Experiments
  6. Experimental Setup
  7. Results Analysis
  8. Conclusion
  9. Illustration of long-tailed data
  10. Key lemmas
  11. Training Loss analysis
  12. Network Gradient
  13. Bound of the Clipping Multiplier $h(C, \mathbf{x}, y)$
  14. Training Loss Bounds
  15. Proof of Theorem \ref{['thm:mem']}
  16. ...and 5 more sections

Key Result

Theorem 1

Under Condition condition and Assumption ass:non-perfect, for any $(\mathbf{x}, y)$ in the training dataset $\mathcal{S}$, after $T \geq \Omega \left( (\eta \Lambda_y \|\mathbf{A}_y\|_F)^{-1} n \sqrt{m} \sigma_0 \right)$ iterations, with probability at least $1 - \delta$, the inner product between t

Figures (4)

  • Figure 1: Heatmap of test accuracy on synthetic data across various feature strength and noise correlation ratio
  • Figure 2: Test Accuracy across Top and Bottom Influence Score Quantiles under DP and Non-DP
  • Figure 3: Training Dynamics Under DP and Non-DP
  • Figure 4: Illustration of long-tailed data on MNIST

Theorems & Definitions (55)

  • Definition 1: Data Generation Model
  • Definition 2: ($\epsilon, \delta_{DP}$)-Differential privacy.
  • Theorem 1: noise pattern memorization
  • Remark 1
  • Definition 3: $L$-Long-tailed data set
  • Theorem 2: Training loss
  • Remark 2
  • Theorem 3: Test error
  • Remark 3: Disproportionate Impact of DP-SGD on Long-Tailed Data
  • Remark 4: Privacy-Utility Trade-off
  • ...and 45 more