Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Differentially Private Gradient-Tracking-Based Distributed Stochastic Optimization over Directed Graphs

Jialong Chen, Jimin Wang, Ji-Feng Zhang

TL;DR

The paper introduces a differentially private gradient-tracking algorithm for distributed stochastic optimization over directed graphs, using two schemes that couple step-size schedules with adaptive subsampling to achieve a finite cumulative privacy budget even over infinite iterations. It proves almost sure and mean-square convergence for nonconvex objectives, with polynomial rates under PL conditions for Scheme (S1) and exponential rates for Scheme (S2). The work also provides a rigorous privacy analysis, showing finite DP budgets and detailing the privacy–convergence trade-off, supported by numerical experiments on MNIST and CIFAR-10 that illustrate practical performance and privacy guarantees. This framework broadens the applicability of differentially private distributed optimization to directed networks and nonconvex objectives while quantifying the privacy成本與效益的折衷.

Abstract

This paper proposes a differentially private gradient-tracking-based distributed stochastic optimization algorithm over directed graphs. In particular, privacy noises are incorporated into each agent's state and tracking variable to mitigate information leakage, after which the perturbed states and tracking variables are transmitted to neighbors. We design two novel schemes for the step-sizes and the sampling number within the algorithm. The sampling parameter-controlled subsampling method employed by both schemes enhances the differential privacy level, and ensures a finite cumulative privacy budget even over infinite iterations. The algorithm achieves both almost sure and mean square convergence for nonconvex objectives. Furthermore, when nonconvex objectives satisfy the Polyak-Lojasiewicz (PL) condition, Scheme (S1) achieves a polynomial mean square convergence rate, and Scheme (S2) achieves an exponential mean square convergence rate. The trade-off between privacy and convergence is presented. The effectiveness of the algorithm and its superior performance compared to existing works are illustrated through numerical examples of distributed training on the benchmark datasets "MNIST" and "CIFAR-10".

Differentially Private Gradient-Tracking-Based Distributed Stochastic Optimization over Directed Graphs

TL;DR

The paper introduces a differentially private gradient-tracking algorithm for distributed stochastic optimization over directed graphs, using two schemes that couple step-size schedules with adaptive subsampling to achieve a finite cumulative privacy budget even over infinite iterations. It proves almost sure and mean-square convergence for nonconvex objectives, with polynomial rates under PL conditions for Scheme (S1) and exponential rates for Scheme (S2). The work also provides a rigorous privacy analysis, showing finite DP budgets and detailing the privacy–convergence trade-off, supported by numerical experiments on MNIST and CIFAR-10 that illustrate practical performance and privacy guarantees. This framework broadens the applicability of differentially private distributed optimization to directed networks and nonconvex objectives while quantifying the privacy成本與效益的折衷.

Abstract

This paper proposes a differentially private gradient-tracking-based distributed stochastic optimization algorithm over directed graphs. In particular, privacy noises are incorporated into each agent's state and tracking variable to mitigate information leakage, after which the perturbed states and tracking variables are transmitted to neighbors. We design two novel schemes for the step-sizes and the sampling number within the algorithm. The sampling parameter-controlled subsampling method employed by both schemes enhances the differential privacy level, and ensures a finite cumulative privacy budget even over infinite iterations. The algorithm achieves both almost sure and mean square convergence for nonconvex objectives. Furthermore, when nonconvex objectives satisfy the Polyak-Lojasiewicz (PL) condition, Scheme (S1) achieves a polynomial mean square convergence rate, and Scheme (S2) achieves an exponential mean square convergence rate. The trade-off between privacy and convergence is presented. The effectiveness of the algorithm and its superior performance compared to existing works are illustrated through numerical examples of distributed training on the benchmark datasets "MNIST" and "CIFAR-10".
Paper Structure (23 sections, 13 theorems, 163 equations, 8 figures, 1 table, 1 algorithm)

This paper contains 23 sections, 13 theorems, 163 equations, 8 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries and problem formulation
  3. Graph theory
  4. Problem formulation
  5. Local differential privacy
  6. Main results
  7. The proposed algorithm
  8. Convergence analysis
  9. Privacy analysis
  10. Trade-off between privacy and convergence
  11. Numerical examples
  12. Effect of privacy noises
  13. Comparison between Schemes (S1) and (S2)
  14. Comparison with methods in kang2021weightedwang2023quantizationyan2024killingwang2024differentiallyachen2024differentially
  15. Conclusion
  16. ...and 8 more sections

Key Result

Lemma 1

If Assumption asm1 holds, then following statements hold: (i) Let $\{\varpi_1^{(1)}\!\!\!,\dots,\varpi_n^{(1)}\!\}$ be the eigenvalues of the matrix $\mathcal{L}_1$ such that $|\varpi_1^{(1)}|\leq\dots\leq|\varpi_n^{(1)}|$, and $\{\varpi_1^{(2)}\!\!\!,\dots,\varpi_n^{(2)}\!\}$ be the eigenvalues of Then, there exist unique nonnegative vectors $v_1$, $v_2\in\mathbb{R}^n$ such that $v_1^\top(I_n-\a

Figures (8)

  • Figure 1: Topology structures of directed graphs $\mathcal{G}_{\mathcal{R}},\mathcal{G}_{\mathcal{C}}$ induced by weight matrices $\mathcal{R},\mathcal{C}$
  • Figure 2: Accuracy and cumulative differential privacy budget $\varepsilon$ of Algorithm \ref{['algorithm1']} with Scheme (S1) and $p_{\zeta_i},p_{\eta_i}=-0.1,0.1,0.2$
  • Figure 3: Accuracy and cumulative differential privacy budget $\varepsilon$ of Algorithm \ref{['algorithm1']} with Scheme (S2) and $p_{\zeta_i},p_{\eta_i}=0.9994,0.9996,0.9998$
  • Figure 4: Comparison of Algorithm \ref{['algorithm1']} with Schemes (S1), (S2) on accuracy and cumulative differential privacy budget $\varepsilon$
  • Figure 5: Accuracy and cumulative differential privacy budget $\varepsilon$ of Algorithm \ref{['algorithm1']} with Scheme (S1) over directed graphs with and without self-loops
  • ...and 3 more figures

Theorems & Definitions (42)

  • Remark 1
  • Lemma 1
  • Remark 2
  • Remark 3
  • Remark 4
  • Definition 1
  • Definition 2
  • Definition 3
  • Remark 5
  • Remark 6
  • ...and 32 more