Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Hubs and Spokes Learning: Efficient and Scalable Collaborative Machine Learning

Atul Sharma, Kavindu Herath, Saurabh Bagchi, Chaoyue Liu, Somali Chaterji

TL;DR

Hubs and Spokes Learning (HSL) offers a scalable, resilient framework that blends Federated Learning's connectivity with decentralized Peer-to-Peer Learning's (P2PL) robustness by organizing nodes into spokes and hubs. The three-stage communication protocol—spoke-to-hub push, hub gossip, and hub-to-spoke pull—enables efficient model propagation via the end-to-end mixing matrix W_{hsl} = W_{sh} W_{hh} W_{hs}, with budgets that can be tuned to balance load and performance. The authors provide non-asymptotic convergence guarantees under standard smoothness, stochastic noise, and heterogeneity assumptions, plus explicit consensus-distance bounds that quantify mixing efficiency through stage-wise betas. Empirically, HSL consistently outperforms Epidemic Learning Local (ELL) at equal budgets and matches ELL with substantially fewer edges on CIFAR-10 and AG News, supported by spectral-gap analyses showing stronger mixing. Collectively, these results position HSL as a practical, scalable alternative that bridges FL and fully decentralized approaches for large-scale distributed learning across edge devices and heterogeneous data distributions.

Abstract

We introduce the Hubs and Spokes Learning (HSL) framework, a novel paradigm for collaborative machine learning that combines the strengths of Federated Learning (FL) and Decentralized Learning (P2PL). HSL employs a two-tier communication structure that avoids the single point of failure inherent in FL and outperforms the state-of-the-art P2PL framework, Epidemic Learning Local (ELL). At equal communication budgets (total edges), HSL achieves higher performance than ELL, while at significantly lower communication budgets, it can match ELL's performance. For instance, with only 400 edges, HSL reaches the same test accuracy that ELL achieves with 1000 edges for 100 peers (spokes) on CIFAR-10, demonstrating its suitability for resource-constrained systems. HSL also achieves stronger consensus among nodes after mixing, resulting in improved performance with fewer training rounds. We substantiate these claims through rigorous theoretical analyses and extensive experimental results, showcasing HSL's practicality for large-scale collaborative learning.

Hubs and Spokes Learning: Efficient and Scalable Collaborative Machine Learning

TL;DR

Hubs and Spokes Learning (HSL) offers a scalable, resilient framework that blends Federated Learning's connectivity with decentralized Peer-to-Peer Learning's (P2PL) robustness by organizing nodes into spokes and hubs. The three-stage communication protocol—spoke-to-hub push, hub gossip, and hub-to-spoke pull—enables efficient model propagation via the end-to-end mixing matrix W_{hsl} = W_{sh} W_{hh} W_{hs}, with budgets that can be tuned to balance load and performance. The authors provide non-asymptotic convergence guarantees under standard smoothness, stochastic noise, and heterogeneity assumptions, plus explicit consensus-distance bounds that quantify mixing efficiency through stage-wise betas. Empirically, HSL consistently outperforms Epidemic Learning Local (ELL) at equal budgets and matches ELL with substantially fewer edges on CIFAR-10 and AG News, supported by spectral-gap analyses showing stronger mixing. Collectively, these results position HSL as a practical, scalable alternative that bridges FL and fully decentralized approaches for large-scale distributed learning across edge devices and heterogeneous data distributions.

Abstract

We introduce the Hubs and Spokes Learning (HSL) framework, a novel paradigm for collaborative machine learning that combines the strengths of Federated Learning (FL) and Decentralized Learning (P2PL). HSL employs a two-tier communication structure that avoids the single point of failure inherent in FL and outperforms the state-of-the-art P2PL framework, Epidemic Learning Local (ELL). At equal communication budgets (total edges), HSL achieves higher performance than ELL, while at significantly lower communication budgets, it can match ELL's performance. For instance, with only 400 edges, HSL reaches the same test accuracy that ELL achieves with 1000 edges for 100 peers (spokes) on CIFAR-10, demonstrating its suitability for resource-constrained systems. HSL also achieves stronger consensus among nodes after mixing, resulting in improved performance with fewer training rounds. We substantiate these claims through rigorous theoretical analyses and extensive experimental results, showcasing HSL's practicality for large-scale collaborative learning.
Paper Structure (33 sections, 7 theorems, 159 equations, 6 figures, 1 algorithm)

This paper contains 33 sections, 7 theorems, 159 equations, 6 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Hubs and Spokes Learning (HSL):
  3. Background and Related Work
  4. Federated Learning (FL):
  5. Peer-to-Peer Learning (P2PL):
  6. Epidemic Learning (EL):
  7. Other Approaches:
  8. Hubs and Spokes Learning (HSL)
  9. Training in HSL:
  10. Theoretical Analysis
  11. Convergence of HSL
  12. Consensus Distance Ratio in HSL
  13. Evaluation
  14. Experimental Setup:
  15. Comparison with ELL Under Varying Budgets:
  16. ...and 18 more sections

Key Result

Theorem 4.4

Consider Algorithm alg:hsl under the above assumptions. Let the initial optimization gap be: Then, for any $T \geq 1$, with $n_s \geq 2$ spokes, a communication budget of $b_{sh} \geq 1$, and $n_h \geq 2$ hubs with budgets $b_{hs} \geq 1, b_{hh} \geq 1$, selecting the step size as: we have where and

Figures (6)

  • Figure 1: A snapshot of the Hubs and Spokes Learning (HSL) network with 9 spokes and 4 hubs with 25 directed edges, where connections dynamically change in each round, illustrating the three-stage communication process. In Stage 1 (Spoke-to-Hub Push), hubs aggregate models from $b_{hs} = 3$ randomly sampled spokes (depicted by blue dashed lines). In Stage 2 (Hub Gossip), each hub exchanges models with $b_{hh} = 1$ other hub (shown in red dashed lines) and averages the received models along with its own. Finally, in Stage 3 (Hub-to-Spoke Pull), each spoke retrieves a model from $b_{sh} = 1$ randomly selected hub (shown by purple dotted lines).
  • Figure 2: HSL vs. ELL on CIFAR-10 ($n_s=100, 200$). (Top) $n_s=100$: (Left) Final accuracy vs. total edges (budget). Candle bodies represent the interquartile range, and wicks indicate min/max values. HSL consistently achieves higher accuracy at lower budgets, demonstrating its efficiency and scalability. (Right) Mixing efficiency over 500 rounds, measured via $-\log(\text{CDR})$, where higher values indicate stronger mixing. HSL achieves superior mixing at all budgets, explaining its improved accuracy. (Bottom) $n_s=200$: HSL maintains its advantage, matching ELL’s 3000-edge accuracy with only a third of the budget. The CDR plot further highlights HSL’s superior mixing, where HSL with just 595 edges achieves better consensus than ELL with 3000 edges, reinforcing its scalability.
  • Figure 3: HSL vs. ELL on AG News ($n_s=100$). The left plot shows the final accuracy distribution, while the right plot presents the consensus distance ratio (CDR). HSL with only 400 edges matches ELL's performance with 1000 edges. The CDR plot continues to confirm the superior mixing efficiency of HSL over ELL.
  • Figure 4: Test Accuracy vs. Training Rounds on AG News ($n_s=100$). We compare FedAvg (200 edges) with HSL and other decentralized methods (400 edges). HSL consistently outperforms decentralized baselines and closely follows FedAvg. Torus reaches 70% accuracy, serving as a baseline, while EL Local and Erdős-Rényi exhibit similar trends due to their dynamic graphs.
  • Figure 5: Average spectral gap variation of HSL, ELL, and Erdős-Rényi with total directed edges. The spectral gap was computed from the effective mixing matrix, sampled at each round for 1000 rounds with 100 spokes, and then averaged. Erdős-Rényi serves as a reference baseline for comparison. The results reaffirm HSL’s superior mixing efficiency, even in mathematical simulations.
  • ...and 1 more figures

Theorems & Definitions (19)

  • Theorem 4.4
  • Lemma 1.1
  • Lemma 1.2
  • Lemma 1.3
  • Lemma 1.4
  • Lemma 1.5
  • Lemma 1.6
  • proof : Proof
  • Remark 3.1
  • Remark 3.2
  • ...and 9 more