Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FedADMM-InSa: An Inexact and Self-Adaptive ADMM for Federated Learning

Yongcun Song, Ziqi Wang, Enrique Zuazua

TL;DR

This work tackles federated learning under data and system heterogeneity by proposing FedADMM-InSa, an ADMM-based method with an inexact local-update criterion and a self-adaptive penalty mechanism. The inexactness criterion lets clients control local computation without fixed epochs, while the adaptive penalties balance primal/dual residuals and improve robustness. The authors prove convergence under strong convexity for the inexact version and validate the approach through extensive experiments (linear regression, MNIST, CIFAR-10), showing substantial reductions in local computation and faster convergence compared to vanilla FedADMM and FedAvg. The results indicate practical impact for resource-constrained FL deployments and point to future work on privacy enhancements and scalability.

Abstract

Federated learning (FL) is a promising framework for learning from distributed data while maintaining privacy. The development of efficient FL algorithms encounters various challenges, including heterogeneous data and systems, limited communication capacities, and constrained local computational resources. Recently developed FedADMM methods show great resilience to both data and system heterogeneity. However, they still suffer from performance deterioration if the hyperparameters are not carefully tuned. To address this issue, we propose an inexact and self-adaptive FedADMM algorithm, termed FedADMM-InSa. First, we design an inexactness criterion for the clients' local updates to eliminate the need for empirically setting the local training accuracy. This inexactness criterion can be assessed by each client independently based on its unique condition, thereby reducing the local computational cost and mitigating the undesirable straggle effect. The convergence of the resulting inexact ADMM is proved under the assumption of strongly convex loss functions. Additionally, we present a self-adaptive scheme that dynamically adjusts each client's penalty parameter, enhancing algorithm robustness by mitigating the need for empirical penalty parameter choices for each client. Extensive numerical experiments on both synthetic and real-world datasets are conducted. As validated by some numerical tests, our proposed algorithm can reduce the clients' local computational load significantly and also accelerate the learning process compared to the vanilla FedADMM.

FedADMM-InSa: An Inexact and Self-Adaptive ADMM for Federated Learning

TL;DR

This work tackles federated learning under data and system heterogeneity by proposing FedADMM-InSa, an ADMM-based method with an inexact local-update criterion and a self-adaptive penalty mechanism. The inexactness criterion lets clients control local computation without fixed epochs, while the adaptive penalties balance primal/dual residuals and improve robustness. The authors prove convergence under strong convexity for the inexact version and validate the approach through extensive experiments (linear regression, MNIST, CIFAR-10), showing substantial reductions in local computation and faster convergence compared to vanilla FedADMM and FedAvg. The results indicate practical impact for resource-constrained FL deployments and point to future work on privacy enhancements and scalability.

Abstract

Federated learning (FL) is a promising framework for learning from distributed data while maintaining privacy. The development of efficient FL algorithms encounters various challenges, including heterogeneous data and systems, limited communication capacities, and constrained local computational resources. Recently developed FedADMM methods show great resilience to both data and system heterogeneity. However, they still suffer from performance deterioration if the hyperparameters are not carefully tuned. To address this issue, we propose an inexact and self-adaptive FedADMM algorithm, termed FedADMM-InSa. First, we design an inexactness criterion for the clients' local updates to eliminate the need for empirically setting the local training accuracy. This inexactness criterion can be assessed by each client independently based on its unique condition, thereby reducing the local computational cost and mitigating the undesirable straggle effect. The convergence of the resulting inexact ADMM is proved under the assumption of strongly convex loss functions. Additionally, we present a self-adaptive scheme that dynamically adjusts each client's penalty parameter, enhancing algorithm robustness by mitigating the need for empirical penalty parameter choices for each client. Extensive numerical experiments on both synthetic and real-world datasets are conducted. As validated by some numerical tests, our proposed algorithm can reduce the clients' local computational load significantly and also accelerate the learning process compared to the vanilla FedADMM.
Paper Structure (42 sections, 3 theorems, 67 equations, 5 figures, 2 tables, 3 algorithms)

This paper contains 42 sections, 3 theorems, 67 equations, 5 figures, 2 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Current Challenges
  4. Research Motivations
  5. Main Contributions
  6. Organization
  7. Preliminaries
  8. Federated Learning
  9. ADMM
  10. Vanilla FedADMM
  11. Client's Local Update: Subproblems \ref{['fedadmm_vanilla_u']} and \ref{['fedadmm_vanilla_l']}
  12. Server's Aggregation: Subproblem \ref{['fedadmm_vanilla_z']}
  13. An Inexact and Self-Adaptive FedADMM
  14. Server's Aggregation with Memory
  15. An Inexact Version of FedADMM
  16. ...and 27 more sections

Key Result

Theorem 3.3

Let $\{w^k\}=\{(u^k,\lambda^k, z^k)^{\top}\}$ be the sequence generated by iterative scheme fedadmm_in (i.e., alg_FedADMM_in with full client participation). Then, we have the following assertions:

Figures (5)

  • Figure 1: Comparison results of FedADMM-In in three examples.
  • Figure 2: Comparison results of FedADMM-InSa in Example 1 (Linear regression).
  • Figure 3: Comparison results of FedADMM-InSa in Example 2 (MNIST with CNN).
  • Figure 4: Comparison results of FedADMM-InSa in Example 3 (CIFAR-10 with ResNet).
  • Figure 5: Comparison of FedADMM and FedADMM-InSa with different values of $\beta_i$ in Example 2. FedADMM (top row) uses fixed penalty parameters. Our FedADMM-InSa (bottom row) utilizes adaptive penalty parameter scheme \ref{['self_adp_beta_rule']}.

Theorems & Definitions (7)

  • Theorem 3.3
  • Remark 3.4
  • Lemma A.1
  • proof
  • Lemma A.2
  • proof
  • proof : Proof of \ref{['thm_fedadmmIn']}