Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Sporadic Gradient Tracking over Directed Graphs: A Theoretical Perspective on Decentralized Federated Learning

Shahryar Zehtabi, Dong-Jun Han, Seyyedali Hosseinalipour, Christopher Brinton

TL;DR

This work addresses decentralized federated learning on directed graphs under heterogeneous computation and communication constraints. It introduces Sporadic Gradient Tracking (Spod-GT), a gradient-tracking based algorithm that allows client-specific gradient computation frequencies and edge activation, enabling sporadic participation while preserving convergence guarantees for non-convex losses. The authors establish a convergence theory with relaxed gradient variance and diversity assumptions, proving consensus and optimality despite intermittent updates, and they provide explicit learning-rate and participation constraints. Empirical results on Fashion-MNIST and CIFAR-10 show that Spod-GT consistently outperforms strong GT baselines in terms of accuracy versus total delay, validating the practical benefits of sporadic computation and communication in directed-network DFL.

Abstract

Decentralized Federated Learning (DFL) enables clients with local data to collaborate in a peer-to-peer manner to train a generalized model. In this paper, we unify two branches of work that have separately solved important challenges in DFL: (i) gradient tracking techniques for mitigating data heterogeneity and (ii) accounting for diverse availability of resources across clients. We propose $\textit{Sporadic Gradient Tracking}$ ($\texttt{Spod-GT}$), the first DFL algorithm that incorporates these factors over general directed graphs by allowing (i) client-specific gradient computation frequencies and (ii) heterogeneous and asymmetric communication frequencies. We conduct a rigorous convergence analysis of our methodology with relaxed assumptions on gradient estimation variance and gradient diversity of clients, providing consensus and optimality guarantees for GT over directed graphs despite intermittent client participation. Through numerical experiments on image classification datasets, we demonstrate the efficacy of $\texttt{Spod-GT}$ compared to well-known GT baselines.

Sporadic Gradient Tracking over Directed Graphs: A Theoretical Perspective on Decentralized Federated Learning

TL;DR

This work addresses decentralized federated learning on directed graphs under heterogeneous computation and communication constraints. It introduces Sporadic Gradient Tracking (Spod-GT), a gradient-tracking based algorithm that allows client-specific gradient computation frequencies and edge activation, enabling sporadic participation while preserving convergence guarantees for non-convex losses. The authors establish a convergence theory with relaxed gradient variance and diversity assumptions, proving consensus and optimality despite intermittent updates, and they provide explicit learning-rate and participation constraints. Empirical results on Fashion-MNIST and CIFAR-10 show that Spod-GT consistently outperforms strong GT baselines in terms of accuracy versus total delay, validating the practical benefits of sporadic computation and communication in directed-network DFL.

Abstract

Decentralized Federated Learning (DFL) enables clients with local data to collaborate in a peer-to-peer manner to train a generalized model. In this paper, we unify two branches of work that have separately solved important challenges in DFL: (i) gradient tracking techniques for mitigating data heterogeneity and (ii) accounting for diverse availability of resources across clients. We propose (), the first DFL algorithm that incorporates these factors over general directed graphs by allowing (i) client-specific gradient computation frequencies and (ii) heterogeneous and asymmetric communication frequencies. We conduct a rigorous convergence analysis of our methodology with relaxed assumptions on gradient estimation variance and gradient diversity of clients, providing consensus and optimality guarantees for GT over directed graphs despite intermittent client participation. Through numerical experiments on image classification datasets, we demonstrate the efficacy of compared to well-known GT baselines.
Paper Structure (38 sections, 15 theorems, 76 equations, 5 figures, 3 tables, 1 algorithm)

This paper contains 38 sections, 15 theorems, 76 equations, 5 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Problem Formulation
  5. Sporadic Gradient Tracking
  6. Matrix Representation
  7. Convergence Analysis
  8. Assumptions and Immediate Implications
  9. Definitions
  10. Propositions
  11. Main Theorem
  12. Experiments
  13. Setup
  14. Results and Discussion
  15. Conclusion
  16. ...and 23 more sections

Key Result

Proposition 4.8

Let Assumptions assump:lipschitz-assump:graph hold, and the learning rates $\eta_i^{(k)}$ satisfy then the spectral radius of $\boldsymbol{\Psi}^{(k)}$ defined in Eq. eqn:varsigma is strictly less than one, i.e., $\rho(\boldsymbol{\Psi}^{(k)}) < 1$. The values of $\kappa_3$, $\kappa_4$, $\kappa_5$, $\hat{\rho}_{0,A}$, $\tilde{\rho}_A$ and $\tilde{\rho}_B$ are given in Table tab:symbols_lemmas in

Figures (5)

  • Figure 1: Illustration of GT-enhanced DFL over directed graphs with clients having heterogeneous computation and communication capabilities. While in conventional GT, local updates and inter-client communications occur at every iteration of training, Spod-GT applies sporadicity in both communications and computations (dashed lines, thickness representing relative frequency).
  • Figure 2: Accuracy vs. latency plots. Spod-GT achieves the target accuracy much faster with less delay, emphasizing the benefit of sporadicity in DFL over directed graphs for GT iterations and aggregations simultaneously.
  • Figure 3: Effects of system parameters on FMNIST. The overall results confirm the advantage of Spod-GT.
  • Figure 4: A schematic of the sequence of our theoretical results and their dependence on each other.
  • Figure 5: Effects of system parameters on CIFAR-10. The overall results confirm the advantage of Spod-GT.

Theorems & Definitions (29)

  • Definition 4.6
  • Definition 4.7: Consensus Error
  • Proposition 4.8: Spectral Radius of Transition Matrix
  • proof
  • Proposition 4.9: Loss Descent Constraints
  • proof
  • Theorem 4.10: Convergence for Non-Convex Loss
  • proof
  • Corollary 4.11: Arbitrarily Small Bound
  • Lemma 2.1: Sporadic Gradient Tracking Discrepancy
  • ...and 19 more