Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Privacy Power of Correlated Noise in Decentralized Learning

Youssef Allouah, Anastasia Koloskova, Aymane El Firdoussi, Martin Jaggi, Rachid Guerraoui

TL;DR

The paper addresses privacy in decentralized learning by introducing Decor, a variant of decentralized SGD that injects two forms of Gaussian noise—pairwise-correlated noise generated from shared secrets and uncorrelated noise for gossip averaging—to protect local models under SecLDP. It shows that, for any connected graph, Decor can match the central DP optimal privacy-utility trade-off within the SecLDP framework, with a privacy bound that scales with graph algebraic connectivity and network topology; a practical SecLDP privacy accountant is provided. The authors derive convergence guarantees under $L$-smooth (and PL) assumptions, establish a linear speedup in the number of users, and quantify the impact of correlated noise on the convergence via a graph-dependent slowdown term. Empirically, Decor closely tracks CDP performance and outperforms LDP across strongly convex, non-convex, and real-data tasks on various graph topologies, validating both the privacy and utility benefits and demonstrating practical applicability in decentralized settings.

Abstract

Decentralized learning is appealing as it enables the scalable usage of large amounts of distributed data and resources (without resorting to any central entity), while promoting privacy since every user minimizes the direct exposure of their data. Yet, without additional precautions, curious users can still leverage models obtained from their peers to violate privacy. In this paper, we propose Decor, a variant of decentralized SGD with differential privacy (DP) guarantees. Essentially, in Decor, users securely exchange randomness seeds in one communication round to generate pairwise-canceling correlated Gaussian noises, which are injected to protect local models at every communication round. We theoretically and empirically show that, for arbitrary connected graphs, Decor matches the central DP optimal privacy-utility trade-off. We do so under SecLDP, our new relaxation of local DP, which protects all user communications against an external eavesdropper and curious users, assuming that every pair of connected users shares a secret, i.e., an information hidden to all others. The main theoretical challenge is to control the accumulation of non-canceling correlated noise due to network sparsity. We also propose a companion SecLDP privacy accountant for public use.

The Privacy Power of Correlated Noise in Decentralized Learning

TL;DR

The paper addresses privacy in decentralized learning by introducing Decor, a variant of decentralized SGD that injects two forms of Gaussian noise—pairwise-correlated noise generated from shared secrets and uncorrelated noise for gossip averaging—to protect local models under SecLDP. It shows that, for any connected graph, Decor can match the central DP optimal privacy-utility trade-off within the SecLDP framework, with a privacy bound that scales with graph algebraic connectivity and network topology; a practical SecLDP privacy accountant is provided. The authors derive convergence guarantees under -smooth (and PL) assumptions, establish a linear speedup in the number of users, and quantify the impact of correlated noise on the convergence via a graph-dependent slowdown term. Empirically, Decor closely tracks CDP performance and outperforms LDP across strongly convex, non-convex, and real-data tasks on various graph topologies, validating both the privacy and utility benefits and demonstrating practical applicability in decentralized settings.

Abstract

Decentralized learning is appealing as it enables the scalable usage of large amounts of distributed data and resources (without resorting to any central entity), while promoting privacy since every user minimizes the direct exposure of their data. Yet, without additional precautions, curious users can still leverage models obtained from their peers to violate privacy. In this paper, we propose Decor, a variant of decentralized SGD with differential privacy (DP) guarantees. Essentially, in Decor, users securely exchange randomness seeds in one communication round to generate pairwise-canceling correlated Gaussian noises, which are injected to protect local models at every communication round. We theoretically and empirically show that, for arbitrary connected graphs, Decor matches the central DP optimal privacy-utility trade-off. We do so under SecLDP, our new relaxation of local DP, which protects all user communications against an external eavesdropper and curious users, assuming that every pair of connected users shares a secret, i.e., an information hidden to all others. The main theoretical challenge is to control the accumulation of non-canceling correlated noise due to network sparsity. We also propose a companion SecLDP privacy accountant for public use.
Paper Structure (25 sections, 18 theorems, 98 equations, 2 figures, 4 algorithms)

This paper contains 25 sections, 18 theorems, 98 equations, 2 figures, 4 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Problem Statement
  4. Decor: Decentralized SGD with Correlated Noise
  5. Privacy Analysis
  6. Utility Analysis
  7. Assumptions
  8. Convergence Analysis
  9. Privacy-utility Trade-off
  10. Empirical Evaluation
  11. Strongly Convex Tasks
  12. Non-convex Task
  13. Conclusion
  14. Privacy Analysis
  15. Convergence Analysis
  16. ...and 10 more sections

Key Result

Theorem 0

Let $\alpha > 1$. Each iteration of Decor (Algorithm algo) satisfies $(\alpha, \alpha \varepsilon)$-SecRDP (Definition def:rdp) against where $a_{1}{(\mathcal{G})}$ is the minimum algebraic connectivity across subgraphs obtained by deleting a single vertex from $\mathcal{G}$. Moreover, $\varepsilon$ can be computed numerically with Algorithm algo:account.

Figures (2)

  • Figure 1: Privacy-utility trade-offs for Decor and the CDP and LDP baselines on least-squares regression, logistic regression, and neural network training under $(\varepsilon, 10^{-5})$-SecLDP against an external eavesdropper observing all communications. Decor closely matches the performance of CDP, and considerably surpasses LDP, across all considered tasks, privacy budgets, and topologies.
  • Figure 3: Example-level SecLDP privacy budget $\varepsilon$, using Algorithm \ref{['algo:account']} and RDP amplification by subsampling wang2019subsampled, as function of $\sigma_{\mathrm{cor}}$ given a fixed $\sigma_{\mathrm{cdp}} = \frac{5C}{1000}$, a total number of iterations $T = 1000$, clipping threshold $C = 1$ and batch size $64$.

Theorems & Definitions (31)

  • Definition 1: SecLDP
  • Theorem 0
  • Definition 2: Mixing matrix
  • Theorem 0
  • Corollary 0
  • Definition 3: $\alpha$-Rényi divergence
  • Lemma 1: gil2013renyi
  • Lemma 2
  • Definition 4: SecRDP
  • Theorem 2
  • ...and 21 more