FedGiA: An Efficient Hybrid Algorithm for Federated Learning

TL;DR

FedGiA addresses the dual challenges of communication efficiency and computation cost in federated learning by blending gradient descent with inexact ADMM updates in a mixed-update framework. It introduces a tunable communication schedule through $k_0$ and partitions clients into two groups to perform either inexact-ADMM updates or a single gradient step, achieving reduced communication rounds while maintaining convergence. The authors establish global convergence under mild assumptions and derive a sublinear rate of $O(k_0/k)$ for the gradient-norm, with potentially faster rates under the KL property or strong convexity, and linear convergence in the strongly convex case. Numerical experiments on linear and logistic regression tasks demonstrate that FedGiA attains similar objective values with substantially fewer communication rounds and lower computation times compared to state-of-the-art baselines, highlighting its practical impact for resource-constrained FL settings.

Abstract

Federated learning has shown its advances recently but is still facing many challenges, such as how algorithms save communication resources and reduce computational costs, and whether they converge. To address these critical issues, we propose a hybrid federated learning algorithm (FedGiA) that combines the gradient descent and the inexact alternating direction method of multipliers. The proposed algorithm is more communication- and computation-efficient than several state-of-the-art algorithms theoretically and numerically. Moreover, it also converges globally under mild conditions.

Figures (6)

• Figure 1: Objective function values and errors v.s. iterations. FedGiA$_{\tt G}$ (solid lines) and FedGiA$_{\tt D}$ (dashed lines) solve Example \ref{['ex-linear']} with $m=128$ and $\alpha=0.5$.
• Figure 2: Effect of $k_0$ for FedGiA$_{\tt G}$ (solid lines) and FedGiA$_{\tt D}$ (dashed lines) solving Example \ref{['ex-linear']} with $\alpha=0.5$.
• Figure 3: Effect of $\alpha$. FedGiA$_{\tt G}$ (solid lines) and FedGiA$_{\tt D}$ (dashed lines) solve Example \ref{['ex-linear']} with $m=128$.
• Figure 4: $f(\mathbf{x}^{\tau_{k}})$ v.s. CR for Example \ref{['ex-linear']}.
• Figure 5: $f(\mathbf{x}^{\tau_{k}})$ v.s. CR for Example \ref{['ex-logist']} with qot.

Theorems & Definitions (29)

• Example 2.1: Least square loss
• Example 2.2: $\ell_2$ norm regularized logistic loss
• Definition 2.1
• Lemma 4.1
• Theorem 4.1
• Theorem 4.2
• Remark 4.1
• Corollary 4.1
• Remark 4.2
• Theorem 4.3