Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Scale-Robust Timely Asynchronous Decentralized Learning

Purbesh Mitra, Sennur Ulukus

TL;DR

The paper studies convergence of asynchronous decentralized learning with local SGD and gossip-based model mixing in a fully connected network. It derives a bound on average staleness $\lim_{t\to\infty} E[S^i_j(t)]$ and shows that if per-node gossip rates satisfy $λ_i ∈ Ω(\log n)$, the system attains $O(1)$ staleness (scale-robustness). It also proves that this scaling cannot be achieved by opportunistic gossip schemes, which require $Ω(n)$ gossip capacity. Numerical experiments on linear and nonlinear regression validate the theory, demonstrating robust convergence and the necessity of $Ω(\log n)$ scaling for larger networks, with implications for 6G-era decentralized learning.

Abstract

We consider an asynchronous decentralized learning system, which consists of a network of connected devices trying to learn a machine learning model without any centralized parameter server. The users in the network have their own local training data, which is used for learning across all the nodes in the network. The learning method consists of two processes, evolving simultaneously without any necessary synchronization. The first process is the model update, where the users update their local model via a fixed number of stochastic gradient descent steps. The second process is model mixing, where the users communicate with each other via randomized gossiping to exchange their models and average them to reach consensus. In this work, we investigate the staleness criteria for such a system, which is a sufficient condition for convergence of individual user models. We show that for network scaling, i.e., when the number of user devices $n$ is very large, if the gossip capacity of individual users scales as $Ω(\log n)$, we can guarantee the convergence of user models in finite time. Furthermore, we show that the bounded staleness can only be guaranteed by any distributed opportunistic scheme by $Ω(n)$ scaling.

Scale-Robust Timely Asynchronous Decentralized Learning

TL;DR

The paper studies convergence of asynchronous decentralized learning with local SGD and gossip-based model mixing in a fully connected network. It derives a bound on average staleness and shows that if per-node gossip rates satisfy , the system attains staleness (scale-robustness). It also proves that this scaling cannot be achieved by opportunistic gossip schemes, which require gossip capacity. Numerical experiments on linear and nonlinear regression validate the theory, demonstrating robust convergence and the necessity of scaling for larger networks, with implications for 6G-era decentralized learning.

Abstract

We consider an asynchronous decentralized learning system, which consists of a network of connected devices trying to learn a machine learning model without any centralized parameter server. The users in the network have their own local training data, which is used for learning across all the nodes in the network. The learning method consists of two processes, evolving simultaneously without any necessary synchronization. The first process is the model update, where the users update their local model via a fixed number of stochastic gradient descent steps. The second process is model mixing, where the users communicate with each other via randomized gossiping to exchange their models and average them to reach consensus. In this work, we investigate the staleness criteria for such a system, which is a sufficient condition for convergence of individual user models. We show that for network scaling, i.e., when the number of user devices is very large, if the gossip capacity of individual users scales as , we can guarantee the convergence of user models in finite time. Furthermore, we show that the bounded staleness can only be guaranteed by any distributed opportunistic scheme by scaling.
Paper Structure (5 sections, 3 theorems, 15 equations, 4 figures, 1 algorithm)

This paper contains 5 sections, 3 theorems, 15 equations, 4 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. System Model
  3. Staleness Guarantee
  4. Numerical Results
  5. Conclusion

Key Result

Lemma 1

The expected staleness of a user is bounded as where $\lambda_{min}=\min\{\lambda_1,\lambda_2,\cdots,\lambda_n\}$.

Figures (4)

  • Figure 1: An example of a small distributed learning network. Each device is updating their local ML model via SGD and parallelly, asynchronously gossiping and mixing models with their neighboring devices.
  • Figure 2: Different representations of a decentralized learning system.
  • Figure 3: Loss-epoch plots for linear regression task; $m=1$.
  • Figure 4: Loss-epoch plots for non-linear regression task; $m\sim n$.

Theorems & Definitions (3)

  • Lemma 1
  • Theorem 1
  • Theorem 2