Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Communication-Efficient Federated Learning with Accelerated Client Gradient

Geeho Kim, Jinkyu Kim, Bohyung Han

TL;DR

Federated learning often suffers from slow convergence when client data are heterogeneous and participation is limited. The authors propose FedACG, a simple, communication-efficient framework that broadcasts a momentum-augmented global update using a lookahead gradient and enforces local updates with a momentum-based regularization term, enabling clients to progress along the global gradient trajectory without extra communication or memory. They prove a convergence rate for non-convex objectives and demonstrate substantial accuracy and communication-efficiency gains across CIFAR-10/100, Tiny-ImageNet, and LEAF realistic datasets, particularly at low participation rates. The method is robust to dynamic client sets and compatible with quantization approaches, making it practical for large-scale, real-world federated deployments.

Abstract

Federated learning often suffers from slow and unstable convergence due to the heterogeneous characteristics of participating client datasets. Such a tendency is aggravated when the client participation ratio is low since the information collected from the clients has large variations. To address this challenge, we propose a simple but effective federated learning framework, which improves the consistency across clients and facilitates the convergence of the server model. This is achieved by making the server broadcast a global model with a lookahead gradient. This strategy enables the proposed approach to convey the projected global update information to participants effectively without additional client memory and extra communication costs. We also regularize local updates by aligning each client with the overshot global model to reduce bias and improve the stability of our algorithm. We provide the theoretical convergence rate of our algorithm and demonstrate remarkable performance gains in terms of accuracy and communication efficiency compared to the state-of-the-art methods, especially with low client participation rates. The source code is available at our project page.

Communication-Efficient Federated Learning with Accelerated Client Gradient

TL;DR

Federated learning often suffers from slow convergence when client data are heterogeneous and participation is limited. The authors propose FedACG, a simple, communication-efficient framework that broadcasts a momentum-augmented global update using a lookahead gradient and enforces local updates with a momentum-based regularization term, enabling clients to progress along the global gradient trajectory without extra communication or memory. They prove a convergence rate for non-convex objectives and demonstrate substantial accuracy and communication-efficiency gains across CIFAR-10/100, Tiny-ImageNet, and LEAF realistic datasets, particularly at low participation rates. The method is robust to dynamic client sets and compatible with quantization approaches, making it practical for large-scale, real-world federated deployments.

Abstract

Federated learning often suffers from slow and unstable convergence due to the heterogeneous characteristics of participating client datasets. Such a tendency is aggravated when the client participation ratio is low since the information collected from the clients has large variations. To address this challenge, we propose a simple but effective federated learning framework, which improves the consistency across clients and facilitates the convergence of the server model. This is achieved by making the server broadcast a global model with a lookahead gradient. This strategy enables the proposed approach to convey the projected global update information to participants effectively without additional client memory and extra communication costs. We also regularize local updates by aligning each client with the overshot global model to reduce bias and improve the stability of our algorithm. We provide the theoretical convergence rate of our algorithm and demonstrate remarkable performance gains in terms of accuracy and communication efficiency compared to the state-of-the-art methods, especially with low client participation rates. The source code is available at our project page.
Paper Structure (41 sections, 8 theorems, 35 equations, 8 figures, 11 tables, 1 algorithm)

This paper contains 41 sections, 8 theorems, 35 equations, 8 figures, 11 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Proposed Approach
  4. Preliminaries
  5. Problem setup
  6. FedAvg algorithm
  7. Federated averaging with Accelerated Client Gradient (FedACG)
  8. Accelerated client gradient
  9. Regularization with momentum-integrated model
  10. Discussion
  11. Comparison with FedAvgM
  12. Convergence analysis of FedACG
  13. Experiments
  14. Experimental setup
  15. Datasets
  16. ...and 26 more sections

Key Result

Theorem 1

(Convergence for non-convex functions) Suppose that local functions $\lbrace \mathcal{F}_i \rbrace _{i=1}^{N}$ are non-convex and $L$-smooth. By setting $\eta \leq \frac{(1-\lambda)^2}{64KL(B^2+1)}$, FedACG satisfies where $M_1^2:=\sigma^2+K\left(1-\frac{|S_t|}{N}\right) \sigma_g^2$, $M_2:=\frac{\sigma^2}{K} + \sigma_g^2$, and $D:=\frac{\mathcal{F}\left(\theta^0\right)-\mathcal{F}(\theta^{*})}{1

Figures (8)

  • Figure 1: An illustration of the proposed accelerated client gradient method. We first partially update the global model in the direction of the global momentum (orange) and then aggregate local updates (gray), resulting in the server model in the next round (blue). This anticipatory update aligns individual local updates with the global gradient, achieving speed-up of convergence.
  • Figure 2: Benefit of accelerated client gradient. For FedAvgM, FedCM, and FedACG (without local regularization for fair comparisons) on CIFAR10, we visualize (a) global training loss surfaces with three local models as black circles in the parameter space, (b) weight divergence, and (c) layer-wise CKA values. In (c), the $x$-axis denotes the layer index of ResNet-18 while the $y$-axis corresponds to CKA values measured on the global validation set.
  • Figure A: Training curves of FedACG and FedAvgM on CIFAR-10 in a moderate-scale setting without smoothing.
  • Figure B: The convergence plots of FedACG and the baselines on CIFAR-10 with different federated learning scenarios.
  • Figure C: The convergence plots of FedACG and the baselines on CIFAR-100 with different federated learning scenarios.
  • ...and 3 more figures

Theorems & Definitions (9)

  • Theorem 1
  • Lemma 1: relaxed triangle inequality
  • Lemma 2: sub-linear convergence rate
  • proof
  • Lemma 3: separating mean and variance
  • Theorem 1
  • Lemma 4
  • Lemma 5
  • Lemma 6