Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Dyn-D$^2$P: Dynamic Differentially Private Decentralized Learning with Provable Utility Guarantee

Zehan Zhu, Yan Huang, Xin Wang, Shouling Ji, Jinming Xu

TL;DR

The paper tackles privacy-aware decentralized non-convex optimization under strict DP by introducing Dyn-D$^2$P, which dynamically adjusts gradient clipping bounds $C_k$ and DP noise levels calibrated via Gaussian DP, on time-varying directed graphs. It combines Push-Sum-based mixing with private local SGD and two dynamic variants, Dyn[$C$]-D$^2$P and Dyn[$\mu$]-D$^2$P, plus a fixed-noise baseline Const-D$^2$P, and provides a DP guarantee with a general utility bound demonstrating a $1/\sqrt{n}$ scaling in the number of nodes (up to a clipping-bias term). Theoretical results rely on $L$-smoothness and mixing assumptions to bound the consensus error and establish a trade-off between privacy noise and gradient clipping bias, while experimental results on CIFAR-10 and Fashion-MNIST show Dyn-D$^2$P achieving superior accuracy under strong DP budgets and robustness to hyper-parameters. This work delivers the first utility analysis for dynamic gradient clipping and noise in decentralized non-convex DP and highlights practical privacy-utility gains for distributed learning in directed, time-varying networks.

Abstract

Most existing decentralized learning methods with differential privacy (DP) guarantee rely on constant gradient clipping bounds and fixed-level DP Gaussian noises for each node throughout the training process, leading to a significant accuracy degradation compared to non-private counterparts. In this paper, we propose a new Dynamic Differentially Private Decentralized learning approach (termed Dyn-D$^2$P) tailored for general time-varying directed networks. Leveraging the Gaussian DP (GDP) framework for privacy accounting, Dyn-D$^2$P dynamically adjusts gradient clipping bounds and noise levels based on gradient convergence. This proposed dynamic noise strategy enables us to enhance model accuracy while preserving the total privacy budget. Extensive experiments on benchmark datasets demonstrate the superiority of Dyn-D$^2$P over its counterparts employing fixed-level noises, especially under strong privacy guarantees. Furthermore, we provide a provable utility bound for Dyn-D$^2$P that establishes an explicit dependency on network-related parameters, with a scaling factor of $1/\sqrt{n}$ in terms of the number of nodes $n$ up to a bias error term induced by gradient clipping. To our knowledge, this is the first model utility analysis for differentially private decentralized non-convex optimization with dynamic gradient clipping bounds and noise levels.

Dyn-D$^2$P: Dynamic Differentially Private Decentralized Learning with Provable Utility Guarantee

TL;DR

The paper tackles privacy-aware decentralized non-convex optimization under strict DP by introducing Dyn-DP, which dynamically adjusts gradient clipping bounds and DP noise levels calibrated via Gaussian DP, on time-varying directed graphs. It combines Push-Sum-based mixing with private local SGD and two dynamic variants, Dyn[]-DP and Dyn[]-DP, plus a fixed-noise baseline Const-DP, and provides a DP guarantee with a general utility bound demonstrating a scaling in the number of nodes (up to a clipping-bias term). Theoretical results rely on -smoothness and mixing assumptions to bound the consensus error and establish a trade-off between privacy noise and gradient clipping bias, while experimental results on CIFAR-10 and Fashion-MNIST show Dyn-DP achieving superior accuracy under strong DP budgets and robustness to hyper-parameters. This work delivers the first utility analysis for dynamic gradient clipping and noise in decentralized non-convex DP and highlights practical privacy-utility gains for distributed learning in directed, time-varying networks.

Abstract

Most existing decentralized learning methods with differential privacy (DP) guarantee rely on constant gradient clipping bounds and fixed-level DP Gaussian noises for each node throughout the training process, leading to a significant accuracy degradation compared to non-private counterparts. In this paper, we propose a new Dynamic Differentially Private Decentralized learning approach (termed Dyn-DP) tailored for general time-varying directed networks. Leveraging the Gaussian DP (GDP) framework for privacy accounting, Dyn-DP dynamically adjusts gradient clipping bounds and noise levels based on gradient convergence. This proposed dynamic noise strategy enables us to enhance model accuracy while preserving the total privacy budget. Extensive experiments on benchmark datasets demonstrate the superiority of Dyn-DP over its counterparts employing fixed-level noises, especially under strong privacy guarantees. Furthermore, we provide a provable utility bound for Dyn-DP that establishes an explicit dependency on network-related parameters, with a scaling factor of in terms of the number of nodes up to a bias error term induced by gradient clipping. To our knowledge, this is the first model utility analysis for differentially private decentralized non-convex optimization with dynamic gradient clipping bounds and noise levels.
Paper Structure (21 sections, 9 theorems, 44 equations, 11 figures, 2 tables, 5 algorithms)

This paper contains 21 sections, 9 theorems, 44 equations, 11 figures, 2 tables, 5 algorithms.

Table of Contents

  1. Introduction
  2. Preliminary and Related Work
  3. Algorithm Development
  4. Theoretical Analysis
  5. Experiments
  6. Experimental Setup
  7. Superior Performance against Baseline Methods
  8. Sensitivity to Hyper-parameters
  9. Performance over Different Graphs and Node Numbers
  10. Conclusion
  11. Proof of Theorem \ref{['main_Theorem']}
  12. Bounding accumulative consensus error
  13. Proof of main result in Theorem \ref{['main_Theorem']}
  14. Proof of Corollary \ref{['Corollary_after_theorem']}
  15. Proof of Proposition \ref{['lemma_of_noise_scale']}
  16. ...and 6 more sections

Key Result

Proposition 1

Let $f:\mathcal{D} \rightarrow \mathbb{R}^d$ be a function and $S$ be its $l_2$ sensitivity. Then, drawing a random noise $N$ from Gaussian distribution with $N\thicksim \mathcal{N}\left( 0,\sigma ^2\mathbb{I}_d \right)$ and adding it to $f$ such that $\mathcal{M}(\mathrm{x})=f(\mathrm{x})+N$ satisf where $\mu$ is the privacy budget in the GDP framework, and a smaller value of $\mu$ implies a stro

Figures (11)

  • Figure 1: The evolution of the gradient norm when training ResNet-18 on Cifar-10 dataset in a fully decentralized setting with 4 nodes. It can be observed that the stochastic gradient norm at each node typically decays as training progresses, and eventually falls below the constant clipping bound (indicated by the blue dotted horizontal line) after certain number of iterations, which makes the clipping operation ineffective in the later stages of training.
  • Figure 2: Comparison of convergence performance for five algorithms under $(1, 10^{-4})$-DP guarantee for each node, when training ResNet-18 on Cifar-10 dataset.
  • Figure 3: Comparison of convergence performance for five algorithms under $(3, 10^{-4})$-DP guarantee for each node, when training shallow CNN on FashionMnist dataset.
  • Figure 4: Illustration of the robustness of Dyn-D$^2$P against the values of $\rho_\mu$ and $\rho_c$ under different privacy budget $\epsilon$.
  • Figure 5: Comparison of convergence performance (model utility) for Dyn-D$^2$P over (a) different graphs consisting of 20 nodes and (b) an exponential graph with different number of nodes, respectively.
  • ...and 6 more figures

Theorems & Definitions (20)

  • Definition 1: $(\epsilon,\delta)$-DP dwork2014algorithmic
  • Proposition 1: $\mu$-GDP dong2019gaussian
  • Proposition 2: From $\mu$-GDP to $\left( \epsilon, \delta \right)$-DP bu2020deep
  • Proposition 3: Composition theorem for GDP with varying $\mu_k$ du2021dynamic
  • Proposition 4: Privacy guarantee
  • proof
  • Theorem 1: Model utility
  • proof
  • Remark 1: On clipping bounds $C_k$
  • Corollary 1
  • ...and 10 more