Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FedDuA: Doubly Adaptive Federated Learning

Shokichi Takakura, Seng Pei Liew, Satoshi Hasegawa

TL;DR

This work addresses slow convergence in cross-device federated learning caused by client and gradient heterogeneity. It introduces FedDuA, a global update framework grounded in mirror descent that uses a doubly adaptive step size to capture inter-client and coordinate-wise heterogeneity without increasing client-side cost. The authors prove minimax optimality under an approximate projection condition and derive convergence bounds for convex objectives, demonstrating robustness to hyperparameters and compatibility with partial participation and existing local-update methods like SCAFFOLD. Empirical results across synthetic and real-world tasks show that FedDuA consistently outperforms baselines and effectively handles anisotropic updates, with strong implications for scalable, privacy-preserving distributed learning in practice.

Abstract

Federated learning is a distributed learning framework where clients collaboratively train a global model without sharing their raw data. FedAvg is a popular algorithm for federated learning, but it often suffers from slow convergence due to the heterogeneity of local datasets and anisotropy in the parameter space. In this work, we formalize the central server optimization procedure through the lens of mirror descent and propose a novel framework, called FedDuA, which adaptively selects the global learning rate based on both inter-client and coordinate-wise heterogeneity in the local updates. We prove that our proposed doubly adaptive step-size rule is minimax optimal and provide a convergence analysis for convex objectives. Although the proposed method does not require additional communication or computational cost on clients, extensive numerical experiments show that our proposed framework outperforms baselines in various settings and is robust to the choice of hyperparameters.

FedDuA: Doubly Adaptive Federated Learning

TL;DR

This work addresses slow convergence in cross-device federated learning caused by client and gradient heterogeneity. It introduces FedDuA, a global update framework grounded in mirror descent that uses a doubly adaptive step size to capture inter-client and coordinate-wise heterogeneity without increasing client-side cost. The authors prove minimax optimality under an approximate projection condition and derive convergence bounds for convex objectives, demonstrating robustness to hyperparameters and compatibility with partial participation and existing local-update methods like SCAFFOLD. Empirical results across synthetic and real-world tasks show that FedDuA consistently outperforms baselines and effectively handles anisotropic updates, with strong implications for scalable, privacy-preserving distributed learning in practice.

Abstract

Federated learning is a distributed learning framework where clients collaboratively train a global model without sharing their raw data. FedAvg is a popular algorithm for federated learning, but it often suffers from slow convergence due to the heterogeneity of local datasets and anisotropy in the parameter space. In this work, we formalize the central server optimization procedure through the lens of mirror descent and propose a novel framework, called FedDuA, which adaptively selects the global learning rate based on both inter-client and coordinate-wise heterogeneity in the local updates. We prove that our proposed doubly adaptive step-size rule is minimax optimal and provide a convergence analysis for convex objectives. Although the proposed method does not require additional communication or computational cost on clients, extensive numerical experiments show that our proposed framework outperforms baselines in various settings and is robust to the choice of hyperparameters.
Paper Structure (48 sections, 5 theorems, 55 equations, 8 figures, 6 tables, 1 algorithm)

This paper contains 48 sections, 5 theorems, 55 equations, 8 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Main contributions
  3. Other Related Work
  4. Adaptive Methods
  5. Mirror Descent
  6. Notation
  7. Problem Formulations and Preliminaries
  8. FedAvg
  9. FedOpt
  10. FedExP
  11. Generalized Formulation and Proposed Method
  12. Mirror descent formulation
  13. Proposed Method
  14. Theoretical Analysis
  15. Minimax Optimality
  16. ...and 33 more sections

Key Result

Theorem 3.2

Let $v_t = \bar{\Delta}_t$, $h_t(\eta) := \odv{}{\eta} \phi_t(\theta_t + \eta v_t) - \langle v_t, w_t\rangle$, and $m_t = \frac{1}{2M}\sum_{i=1}^M \|\Delta_i^{t}\|^2$. Assume that A.P.C. in Assumption assumption:approximate-projection-condition holds and $v_t \neq 0$. Then, $\eta_g^* := \arg\min D_{

Figures (8)

  • Figure 1: Test accuracy for $\text{FedDuA}$ and baselines without server momentum (upper) and with server momentum (lower). Our proposed methods (green dashdot) consistently outperform baselines.
  • Figure 2: Test accuracy averaged over the last 5 iterates with different hyperparameters $\epsilon_g, \eta_g$ and $\epsilon$. Proposed methods are less sensitive to the choice of hyperparameters.
  • Figure 3: Performance of $\text{FedDuA}$ with SCAFFOLD.
  • Figure 4: Training of ViT on CIFAR-100.
  • Figure 5: Long term behavior of each algorithm.
  • ...and 3 more figures

Theorems & Definitions (5)

  • Theorem 3.2: Lower Bound on the Optimal Step Size
  • Theorem 3.3: Lower Bound on the Optimal Step Size with Momentum
  • Theorem 4.1: Minimax Optimality
  • Theorem 4.3
  • Corollary 4.4