Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Tailoring Gradient Methods for Differentially-Private Distributed Optimization

Yongqiang Wang, Angelia Nedic

TL;DR

The paper introduces two DP-oriented gradient methods for differentially-private distributed optimization on directed graphs: a static-consensus gradient method and a gradient-tracking (dynamic-consensus) method. It proves that both schemes achieve almost-sure convergence to the global optimum while guaranteeing $\epsilon$-DP with a finite privacy budget, even as the number of iterations grows indefinitely, by carefully weakening inter-agent coupling through time-varying sequences. Theoretical results include a vector-valued martingale framework and convergence/privacy theorems, complemented by numerical experiments on distributed estimation and MNIST-based CNN training that validate practical performance under privacy constraints. The work provides a principled approach to balancing privacy and accuracy in distributed learning, with potential extensions to communication-efficiency and broader network topologies.

Abstract

Decentralized optimization is gaining increased traction due to its widespread applications in large-scale machine learning and multi-agent systems. The same mechanism that enables its success, i.e., information sharing among participating agents, however, also leads to the disclosure of individual agents' private information, which is unacceptable when sensitive data are involved. As differential privacy is becoming a de facto standard for privacy preservation, recently results have emerged integrating differential privacy with distributed optimization. However, directly incorporating differential privacy design in existing distributed optimization approaches significantly compromises optimization accuracy. In this paper, we propose to redesign and tailor gradient methods for differentially-private distributed optimization, and propose two differential-privacy oriented gradient methods that can ensure both rigorous epsilon-differential privacy and optimality. The first algorithm is based on static-consensus based gradient methods, and the second algorithm is based on dynamic-consensus (gradient-tracking) based distributed optimization methods and, hence, is applicable to general directed interaction graph topologies. Both algorithms can simultaneously ensure almost sure convergence to an optimal solution and a finite privacy budget, even when the number of iterations goes to infinity. To our knowledge, this is the first time that both goals are achieved simultaneously. Numerical simulations using a distributed estimation problem and experimental results on a benchmark dataset confirm the effectiveness of the proposed approaches.

Tailoring Gradient Methods for Differentially-Private Distributed Optimization

TL;DR

The paper introduces two DP-oriented gradient methods for differentially-private distributed optimization on directed graphs: a static-consensus gradient method and a gradient-tracking (dynamic-consensus) method. It proves that both schemes achieve almost-sure convergence to the global optimum while guaranteeing -DP with a finite privacy budget, even as the number of iterations grows indefinitely, by carefully weakening inter-agent coupling through time-varying sequences. Theoretical results include a vector-valued martingale framework and convergence/privacy theorems, complemented by numerical experiments on distributed estimation and MNIST-based CNN training that validate practical performance under privacy constraints. The work provides a principled approach to balancing privacy and accuracy in distributed learning, with potential extensions to communication-efficiency and broader network topologies.

Abstract

Decentralized optimization is gaining increased traction due to its widespread applications in large-scale machine learning and multi-agent systems. The same mechanism that enables its success, i.e., information sharing among participating agents, however, also leads to the disclosure of individual agents' private information, which is unacceptable when sensitive data are involved. As differential privacy is becoming a de facto standard for privacy preservation, recently results have emerged integrating differential privacy with distributed optimization. However, directly incorporating differential privacy design in existing distributed optimization approaches significantly compromises optimization accuracy. In this paper, we propose to redesign and tailor gradient methods for differentially-private distributed optimization, and propose two differential-privacy oriented gradient methods that can ensure both rigorous epsilon-differential privacy and optimality. The first algorithm is based on static-consensus based gradient methods, and the second algorithm is based on dynamic-consensus (gradient-tracking) based distributed optimization methods and, hence, is applicable to general directed interaction graph topologies. Both algorithms can simultaneously ensure almost sure convergence to an optimal solution and a finite privacy budget, even when the number of iterations goes to infinity. To our knowledge, this is the first time that both goals are achieved simultaneously. Numerical simulations using a distributed estimation problem and experimental results on a benchmark dataset confirm the effectiveness of the proposed approaches.
Paper Structure (18 sections, 14 theorems, 102 equations, 5 figures, 1 table)

This paper contains 18 sections, 14 theorems, 102 equations, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Problem Formulation and Preliminaries
  3. On distributed optimization
  4. On differential privacy
  5. Static-consensus gradient methods for differentially-private distributed optimization
  6. Convergence analysis
  7. Privacy analysis
  8. gradient-tracking based methods for differentially private distributed optimization
  9. Convergence analysis
  10. Privacy analysis
  11. Numerical Experiments
  12. Evaluation using distributed estimation
  13. Evaluation using image classification on MNIST
  14. Conclusions, Discussions, and Future Work
  15. Proof of Lemma \ref{['th-main_decreasing']}
  16. ...and 3 more sections

Key Result

Lemma 1

Let $\{v^k\}$, $\{u^k\}$, $\{\alpha^k\}$, and $\{{\beta}^k\}$ be random nonnegative scalar sequences such that $\sum_{k=0}^\infty \alpha^k<\infty$ and $\sum_{k=0}^\infty {\beta}^k<\infty$a.s. and where $\mathcal{F}^k=\{v^\ell,u^\ell,\alpha^\ell,{\beta}^\ell;\, 0\le \ell\le k\}$. Then $\sum_{k=0}^{\infty}u^k<\infty$ and $\lim_{k\to\infty}v^k=v$ for a random variable $v\geq 0$a.s.

Figures (5)

  • Figure 1: The interaction topology of the network
  • Figure 2: Comparison of Algorithm 1 with existing distributed gradient descent algorithm (DGD) in nedic2009distributed (under the same noise) and the differential-privacy approach for decentralized optimization PDOP in huang2015differentially (under the same privacy budget) using the distributed estimation problem
  • Figure 3: Comparison of Algorithm 2 with existing dynamic-consensus based distributed gradient algorithm (Push Pull) in pu2020push (under the same noise) and the PDOP-based differential-privacy approach in huang2015differentially for Push Pull (under the same privacy budget) using the distributed estimation problem
  • Figure 4: Comparison of Algorithm 1 with existing distributed gradient descent algorithm (DGD) in nedic2009distributed (under the same noise) and the differential-privacy approach for decentralized optimization PDOP in huang2015differentially (under the same privacy budget) using the MNIST image classification problem
  • Figure 5: Comparison of Algorithm 2 with existing dynamic-consensus based distributed gradient algorithm (Push Pull) in pu2020push (under the same noise) and the PDOP-based differential-privacy approach in huang2015differentially for Push Pull (under the same privacy budget) using the MNIST image classification problem

Theorems & Definitions (40)

  • Lemma 1: polyak87, Lemma 11, page 50
  • Lemma 2
  • proof
  • Lemma 3
  • proof
  • Lemma 4
  • proof
  • Definition 1
  • Remark 1
  • Definition 2
  • ...and 30 more