Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Distributed Federated Learning by Alternating Periods of Training

Shamik Bhattacharyya, Rachel Kalpana Kalaimani

TL;DR

Distributed Federated Learning (DFL) replaces a single central aggregator with a connected network of multiple servers, each serving its own client cohort. It interleaves local gradient updates on clients with inter-server consensus steps to drive all servers to a common parameter while keeping client data on-device to preserve privacy. Under standard $\mu$-strong convexity and $L$-smoothness assumptions and with a step size constraint $\gamma<\min\{1/(L T_C),1/(\mu T_C)\}$, DFL achieves convergence with an error bound $\epsilon$ that depends on the consensus parameter $\sigma_A$, the gradient bound $\theta$, the per-epoch work and initial disagreement, via $\epsilon=\sqrt{M}\gamma\theta T_C\sigma_A/(1-\sigma_A) + Y_0/(1-\Lambda)$ where $\Lambda=\sqrt{1-\gamma \mu T_C}$ and $Y_0$ aggregates initialization terms. Numerical simulations on a data-fitting task validate rapid cross-server consensus, demonstrating the practicality of decentralized federated optimization across distributed data silos with theoretical guarantees. This framework offers a privacy-conscious, scalable approach to federated learning across regional data pools, supported by convergence theory and empirical evidence of effective integration of local and global training.

Abstract

Federated learning is a privacy-focused approach towards machine learning where models are trained on client devices with locally available data and aggregated at a central server. However, the dependence on a single central server is challenging in the case of a large number of clients and even poses the risk of a single point of failure. To address these critical limitations of scalability and fault-tolerance, we present a distributed approach to federated learning comprising multiple servers with inter-server communication capabilities. While providing a fully decentralized approach, the designed framework retains the core federated learning structure where each server is associated with a disjoint set of clients with server-client communication capabilities. We propose a novel DFL (Distributed Federated Learning) algorithm which uses alternating periods of local training on the client data followed by global training among servers. We show that the DFL algorithm, under a suitable choice of parameters, ensures that all the servers converge to a common model value within a small tolerance of the ideal model, thus exhibiting effective integration of local and global training models. Finally, we illustrate our theoretical claims through numerical simulations.

Distributed Federated Learning by Alternating Periods of Training

TL;DR

Distributed Federated Learning (DFL) replaces a single central aggregator with a connected network of multiple servers, each serving its own client cohort. It interleaves local gradient updates on clients with inter-server consensus steps to drive all servers to a common parameter while keeping client data on-device to preserve privacy. Under standard -strong convexity and -smoothness assumptions and with a step size constraint , DFL achieves convergence with an error bound that depends on the consensus parameter , the gradient bound , the per-epoch work and initial disagreement, via where and aggregates initialization terms. Numerical simulations on a data-fitting task validate rapid cross-server consensus, demonstrating the practicality of decentralized federated optimization across distributed data silos with theoretical guarantees. This framework offers a privacy-conscious, scalable approach to federated learning across regional data pools, supported by convergence theory and empirical evidence of effective integration of local and global training.

Abstract

Federated learning is a privacy-focused approach towards machine learning where models are trained on client devices with locally available data and aggregated at a central server. However, the dependence on a single central server is challenging in the case of a large number of clients and even poses the risk of a single point of failure. To address these critical limitations of scalability and fault-tolerance, we present a distributed approach to federated learning comprising multiple servers with inter-server communication capabilities. While providing a fully decentralized approach, the designed framework retains the core federated learning structure where each server is associated with a disjoint set of clients with server-client communication capabilities. We propose a novel DFL (Distributed Federated Learning) algorithm which uses alternating periods of local training on the client data followed by global training among servers. We show that the DFL algorithm, under a suitable choice of parameters, ensures that all the servers converge to a common model value within a small tolerance of the ideal model, thus exhibiting effective integration of local and global training models. Finally, we illustrate our theoretical claims through numerical simulations.
Paper Structure (10 sections, 5 theorems, 32 equations, 3 figures, 1 algorithm)

This paper contains 10 sections, 5 theorems, 32 equations, 3 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. System Model
  4. Distributed Federated Learning
  5. Results
  6. Algorithm
  7. Intermediate Results
  8. Main Result
  9. Numerical Simulation
  10. Conclusion

Key Result

Lemma 1

Suppose Assumptions asmpn:grphCnnctd and asmpn:gradBound hold. Then the DFL algorithm ensures that the difference between the model parameter estimate of any server $i$, $w^i_{p}$ and the global average model parameter estimate across all servers, $\Bar{w}_{p}$ is bounded for every epoch $p$. Specif where $\sigma_A = \Vert A^{T_S} - \frac{1}{M} \mathbf{1} \mathbf{1}' \Vert$ and $\gamma$ is as in e

Figures (3)

  • Figure 1: System model example with M=6 and N=4
  • Figure 2: Timeline representation of client and server iterations over 1 epoch of the DFL algorithm
  • Figure 3: The DFL algorithm (a) generates a best-fitting straight line for the data across all clients, and (b) manages to get all the servers to achieve consensus.

Theorems & Definitions (10)

  • Lemma 1
  • proof
  • Lemma 2
  • proof
  • Lemma 3
  • proof
  • Lemma 4
  • proof
  • Theorem 1
  • proof