Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ADP-VRSGP: Decentralized Learning with Adaptive Differential Privacy via Variance-Reduced Stochastic Gradient Push

Xiaoming Wu, Teng Liu, Xin Wang, Ming Yang, Jiguo Yu

TL;DR

This work tackles privacy-preserving decentralized learning over time-varying directed topologies by introducing ADP-VRSGP, which jointly adapts differential privacy noise and learning rates with a stepwise schedule and reduces gradient variance through progressive gradient fusion. The SDLR module decouples noise decay from learning-rate updates using a piecewise scheme, while PGF leverages historical gradients and adaptive clipping to stabilize training under DP. Theoretical analysis yields improved noise bounds and convergence guarantees in directed networks, with key results showing optimal learning-rate choices under varying noise decay and network dynamics. Empirical results on MNIST, Fashion-MNIST, and CIFAR-10 demonstrate consistent privacy-utility gains over baselines across privacy budgets, network sizes, and data heterogeneity, validating the method’s practicality for privacy-sensitive decentralized learning in dynamic environments.

Abstract

Differential privacy is widely employed in decentralized learning to safeguard sensitive data by introducing noise into model updates. However, existing approaches that use fixed-variance noise often degrade model performance and reduce training efficiency. To address these limitations, we propose a novel approach called decentralized learning with adaptive differential privacy via variance-reduced stochastic gradient push (ADP-VRSGP). This method dynamically adjusts both the noise variance and the learning rate using a stepwise-decaying schedule, which accelerates training and enhances final model performance while providing node-level personalized privacy guarantees. To counteract the slowed convergence caused by large-variance noise in early iterations, we introduce a progressive gradient fusion strategy that leverages historical gradients. Furthermore, ADP-VRSGP incorporates decentralized push-sum and aggregation techniques, making it particularly suitable for time-varying communication topologies. Through rigorous theoretical analysis, we demonstrate that ADP-VRSGP achieves robust convergence with an appropriate learning rate, significantly improving training stability and speed. Experimental results validate that our method outperforms existing baselines across multiple scenarios, highlighting its efficacy in addressing the challenges of privacy-preserving decentralized learning.

ADP-VRSGP: Decentralized Learning with Adaptive Differential Privacy via Variance-Reduced Stochastic Gradient Push

TL;DR

This work tackles privacy-preserving decentralized learning over time-varying directed topologies by introducing ADP-VRSGP, which jointly adapts differential privacy noise and learning rates with a stepwise schedule and reduces gradient variance through progressive gradient fusion. The SDLR module decouples noise decay from learning-rate updates using a piecewise scheme, while PGF leverages historical gradients and adaptive clipping to stabilize training under DP. Theoretical analysis yields improved noise bounds and convergence guarantees in directed networks, with key results showing optimal learning-rate choices under varying noise decay and network dynamics. Empirical results on MNIST, Fashion-MNIST, and CIFAR-10 demonstrate consistent privacy-utility gains over baselines across privacy budgets, network sizes, and data heterogeneity, validating the method’s practicality for privacy-sensitive decentralized learning in dynamic environments.

Abstract

Differential privacy is widely employed in decentralized learning to safeguard sensitive data by introducing noise into model updates. However, existing approaches that use fixed-variance noise often degrade model performance and reduce training efficiency. To address these limitations, we propose a novel approach called decentralized learning with adaptive differential privacy via variance-reduced stochastic gradient push (ADP-VRSGP). This method dynamically adjusts both the noise variance and the learning rate using a stepwise-decaying schedule, which accelerates training and enhances final model performance while providing node-level personalized privacy guarantees. To counteract the slowed convergence caused by large-variance noise in early iterations, we introduce a progressive gradient fusion strategy that leverages historical gradients. Furthermore, ADP-VRSGP incorporates decentralized push-sum and aggregation techniques, making it particularly suitable for time-varying communication topologies. Through rigorous theoretical analysis, we demonstrate that ADP-VRSGP achieves robust convergence with an appropriate learning rate, significantly improving training stability and speed. Experimental results validate that our method outperforms existing baselines across multiple scenarios, highlighting its efficacy in addressing the challenges of privacy-preserving decentralized learning.
Paper Structure (22 sections, 6 theorems, 57 equations, 3 figures, 6 tables, 2 algorithms)

This paper contains 22 sections, 6 theorems, 57 equations, 3 figures, 6 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related work
  3. Preliminary and Problem Formulation
  4. The ADP-VRSGP algorithm
  5. SDLR Strategy
  6. PGF Mechanism
  7. Impact Analysis of the PGF Mechanism
  8. Theoretical Analysis
  9. Experimental Evaluation
  10. Experiments Setup
  11. Performance Comparison with Baselines
  12. Optimal Learning Rate under Varying Noise Decay Rates
  13. Effect of Hyperparamter $\vartheta$ and Time Interval $\tau$
  14. Effect of Node Scalability
  15. Impact of Data Heterogeneity
  16. ...and 7 more sections

Key Result

Lemma 1

If the learning rate coefficient is set as in (learning-rate-coefficient), the bound on the noise error is improved compared to using either $\beta^t = \alpha^t \alpha^{T-t}$ or $\beta^t = \alpha^t \alpha^t$. This adjustment results in a more favorable trade-off between noise reduction and model acc

Figures (3)

  • Figure 1: Test accuracy (%) on the CIFAR-10 dataset under different learning rates.
  • Figure 2: Test accuracy (%) versus hyperparameter $\vartheta$ for different datasets. Upper panels: CNN results; lower panels: ResNet-18 results.
  • Figure 3: Test accuracy (%) on Fashion-MNIST and CIFAR-10 under varying privacy budgets.

Theorems & Definitions (14)

  • Lemma 1
  • proof
  • Lemma 2
  • proof
  • Lemma 3
  • proof
  • Remark 1
  • Theorem 1
  • proof
  • Theorem 2
  • ...and 4 more