Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FedLGA: Towards System-Heterogeneity of Federated Learning via Local Gradient Approximation

Xingyu Li, Zhe Qu, Bo Tang, Zhuo Lu

TL;DR

FedLGA tackles system-heterogeneous federated learning by enabling the aggregator to compensate for uneven local training across devices through local gradient approximation guided by a Hessian-based correction. The method achieves linear-cost Hessian-like updates via an outer-product surrogate and a first-order w_i,E approximation, yielding convergence guarantees for non-convex objectives under both full and partial participation with a device-heterogeneity ratio ρ. Empirically, FedLGA outperforms FedAvg, FedProx, FedNova, Scaffold, and FedDyn on Fashion-MNIST, CIFAR-10, and CIFAR-100 under non-i.i.d. data distributions, achieving faster convergence and better final accuracy while incurring lower total communication rounds. The work demonstrates practical benefits for FL deployments with heterogeneous devices and data distributions, without imposing extra burden on remote devices and with minimal aggregator-side overhead.

Abstract

Federated Learning (FL) is a decentralized machine learning architecture, which leverages a large number of remote devices to learn a joint model with distributed training data. However, the system-heterogeneity is one major challenge in a FL network to achieve robust distributed learning performance, which comes from two aspects: i) device-heterogeneity due to the diverse computational capacity among devices; ii) data-heterogeneity due to the non-identically distributed data across the network. Prior studies addressing the heterogeneous FL issue, e.g., FedProx, lack formalization and it remains an open problem. This work first formalizes the system-heterogeneous FL problem and proposes a new algorithm, called FedLGA, to address this problem by bridging the divergence of local model updates via gradient approximation. To achieve this, FedLGA provides an alternated Hessian estimation method, which only requires extra linear complexity on the aggregator. Theoretically, we show that with a device-heterogeneous ratio $ρ$, FedLGA achieves convergence rates on non-i.i.d. distributed FL training data for the non-convex optimization problems with $\mathcal{O} \left( \frac{(1+ρ)}{\sqrt{ENT}} + \frac{1}{T} \right)$ and $\mathcal{O} \left( \frac{(1+ρ)\sqrt{E}}{\sqrt{TK}} + \frac{1}{T} \right)$ for full and partial device participation respectively, where $E$ is the number of local learning epoch, $T$ is the number of total communication round, $N$ is the total device number and $K$ is the number of selected device in one communication round under partially participation scheme. The results of comprehensive experiments on multiple datasets show that FedLGA outperforms current FL methods against the system-heterogeneity.

FedLGA: Towards System-Heterogeneity of Federated Learning via Local Gradient Approximation

TL;DR

FedLGA tackles system-heterogeneous federated learning by enabling the aggregator to compensate for uneven local training across devices through local gradient approximation guided by a Hessian-based correction. The method achieves linear-cost Hessian-like updates via an outer-product surrogate and a first-order w_i,E approximation, yielding convergence guarantees for non-convex objectives under both full and partial participation with a device-heterogeneity ratio ρ. Empirically, FedLGA outperforms FedAvg, FedProx, FedNova, Scaffold, and FedDyn on Fashion-MNIST, CIFAR-10, and CIFAR-100 under non-i.i.d. data distributions, achieving faster convergence and better final accuracy while incurring lower total communication rounds. The work demonstrates practical benefits for FL deployments with heterogeneous devices and data distributions, without imposing extra burden on remote devices and with minimal aggregator-side overhead.

Abstract

Federated Learning (FL) is a decentralized machine learning architecture, which leverages a large number of remote devices to learn a joint model with distributed training data. However, the system-heterogeneity is one major challenge in a FL network to achieve robust distributed learning performance, which comes from two aspects: i) device-heterogeneity due to the diverse computational capacity among devices; ii) data-heterogeneity due to the non-identically distributed data across the network. Prior studies addressing the heterogeneous FL issue, e.g., FedProx, lack formalization and it remains an open problem. This work first formalizes the system-heterogeneous FL problem and proposes a new algorithm, called FedLGA, to address this problem by bridging the divergence of local model updates via gradient approximation. To achieve this, FedLGA provides an alternated Hessian estimation method, which only requires extra linear complexity on the aggregator. Theoretically, we show that with a device-heterogeneous ratio , FedLGA achieves convergence rates on non-i.i.d. distributed FL training data for the non-convex optimization problems with and for full and partial device participation respectively, where is the number of local learning epoch, is the number of total communication round, is the total device number and is the number of selected device in one communication round under partially participation scheme. The results of comprehensive experiments on multiple datasets show that FedLGA outperforms current FL methods against the system-heterogeneity.
Paper Structure (24 sections, 9 theorems, 44 equations, 8 figures, 5 tables, 2 algorithms)

This paper contains 24 sections, 9 theorems, 44 equations, 8 figures, 5 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Introduction
  3. Background and Problem Formulation
  4. Federated Learning Objective
  5. Problem Formulation
  6. Proposed Algorithm: FedLGA
  7. Design Motivation
  8. Hessian Approximation
  9. Algorithm of FedLGA
  10. Convergence Analysis
  11. Convergence Analysis for Full Participation
  12. Convergence Analysis for Partial Participation
  13. Experiments
  14. Experimental Setup
  15. Analysis of Joint Model Performance
  16. ...and 9 more sections

Key Result

Theorem 1

Let Assumptions Assum:0-Assum:approx hold. The local and global learning rates $\eta_{l}$ and $\eta_{g}$ are chosen such that $\eta_{l} < \frac{1}{\sqrt{30 (1 + \rho)} LE}$ and $\eta_{g} \eta_{l} \leq \frac{1}{(1+\rho) LE}$. Under full device participation scheme, the iterates of FedLGA satisfy where $f^{0} = f (\bm{w}^{0}), f^{\star} = f (\bm{w}^{\star})$, $c_1$ is constant, the expectation is o

Figures (8)

  • Figure 1: The heterogeneous local gradients due to system-heterogeneity of FL in FedAvg, illustrated for $2$ remote devices with three iterations.
  • Figure 2: Learning performance of testing accuracy under the system-heterogeneous FL with $\rho = 0.5, \tau_{max} = E-1$ and $E=5$.
  • Figure 3: Learning performance of training loss under the system-heterogeneous FL with $\rho = 0.5, \tau_{max} = E-1$ and $E=5$.
  • Figure 4: Learning performance of best accuracy under the system-heterogeneous FL with $\rho = 0.5, \tau_{max} = E-1$ and $E=5$.
  • Figure 5: Performance of the compared FL methods under different FL network settings with system-heterogeneity.
  • ...and 3 more figures

Theorems & Definitions (22)

  • Theorem 1
  • proof
  • Corollary 1
  • Remark 1
  • Remark 2
  • Theorem 2
  • proof
  • Corollary 2
  • Remark 3
  • Remark 4
  • ...and 12 more