Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Robust Decentralized Learning with Local Updates and Gradient Tracking

Sajjad Ghiasvand, Amirhossein Reisizadeh, Mahnoosh Alizadeh, Ramtin Pedarsani

TL;DR

The paper addresses robust decentralized learning by formulating a decentralized nonconvex-strongly-concave minimax problem over a connected network and proposing Dec-FedTrack, which combines local updates with gradient tracking to address data heterogeneity and adversarial perturbations. It provides rigorous convergence guarantees, showing that the method achieves an $\epsilon$-stationary point with explicit SFO and communication complexities that scale with the condition number $\kappa=\ell/\mu$ and network connectivity. Theoretical results are complemented by empirical evaluations on robust logistic regression and neural network training, demonstrating improved robustness and communication efficiency against baselines. The work advances federated/minimax learning by enabling effective decentralized training under heterogeneity and adversarial threats, with practical implications for IoT and edge computing deployments.

Abstract

As distributed learning applications such as Federated Learning, the Internet of Things (IoT), and Edge Computing grow, it is critical to address the shortcomings of such technologies from a theoretical perspective. As an abstraction, we consider decentralized learning over a network of communicating clients or nodes and tackle two major challenges: data heterogeneity and adversarial robustness. We propose a decentralized minimax optimization method that employs two important modules: local updates and gradient tracking. Minimax optimization is the key tool to enable adversarial training for ensuring robustness. Having local updates is essential in Federated Learning (FL) applications to mitigate the communication bottleneck, and utilizing gradient tracking is essential to proving convergence in the case of data heterogeneity. We analyze the performance of the proposed algorithm, Dec-FedTrack, in the case of nonconvex-strongly concave minimax optimization, and prove that it converges a stationary point. We also conduct numerical experiments to support our theoretical findings.

Robust Decentralized Learning with Local Updates and Gradient Tracking

TL;DR

The paper addresses robust decentralized learning by formulating a decentralized nonconvex-strongly-concave minimax problem over a connected network and proposing Dec-FedTrack, which combines local updates with gradient tracking to address data heterogeneity and adversarial perturbations. It provides rigorous convergence guarantees, showing that the method achieves an -stationary point with explicit SFO and communication complexities that scale with the condition number and network connectivity. Theoretical results are complemented by empirical evaluations on robust logistic regression and neural network training, demonstrating improved robustness and communication efficiency against baselines. The work advances federated/minimax learning by enabling effective decentralized training under heterogeneity and adversarial threats, with practical implications for IoT and edge computing deployments.

Abstract

As distributed learning applications such as Federated Learning, the Internet of Things (IoT), and Edge Computing grow, it is critical to address the shortcomings of such technologies from a theoretical perspective. As an abstraction, we consider decentralized learning over a network of communicating clients or nodes and tackle two major challenges: data heterogeneity and adversarial robustness. We propose a decentralized minimax optimization method that employs two important modules: local updates and gradient tracking. Minimax optimization is the key tool to enable adversarial training for ensuring robustness. Having local updates is essential in Federated Learning (FL) applications to mitigate the communication bottleneck, and utilizing gradient tracking is essential to proving convergence in the case of data heterogeneity. We analyze the performance of the proposed algorithm, Dec-FedTrack, in the case of nonconvex-strongly concave minimax optimization, and prove that it converges a stationary point. We also conduct numerical experiments to support our theoretical findings.
Paper Structure (21 sections, 9 theorems, 61 equations, 2 figures, 2 tables, 1 algorithm)

This paper contains 21 sections, 9 theorems, 61 equations, 2 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Federated Learning with Heterogeneous Data
  4. Decentralized Minimization
  5. Centralized Minimax Optimization
  6. Decentralized Minimax Optimization
  7. Distributed/Federated Minimax Learning
  8. Problem Setup
  9. Motivating example: Federated adversarial training
  10. Convergence measure
  11. Notation
  12. Proposed Algorithm
  13. Convergence Analysis
  14. Proof Sketch
  15. Empirical Results
  16. ...and 6 more sections

Key Result

Theorem 1

Suppose Assumptions assm: W-assm: lower bound hold and consider the iterates of Dec-FedTrack in Algorithm al.1 with step-sizes $\eta_d=$$\Theta\left(\frac{p}{\kappa K \ell}\right), \eta_c=\Theta\left(\frac{\eta_d}{\kappa^2}\right)$, and $\eta_s=\eta_r=\Theta(p)$. Then, after $T$ communication rounds where $\mathcal{H}_0=O\left(1+\frac{\delta_0}{K \kappa p}\right)$ and $\delta_0 = O\left(\frac{q}{\

Figures (2)

  • Figure 1: Convergence of $\Phi(\bar{\mathbf{x}})$ against the number of SFO calls (above) and the number of communication rounds (bottom).
  • Figure 2: Test accuracy against the number of SFO calls (above) and the number of communication rounds (bottom).

Theorems & Definitions (13)

  • Definition 1
  • Definition 2
  • Theorem 1
  • Remark 1
  • Remark 2
  • Proposition 1
  • Proposition 2
  • Lemma 1
  • Lemma 2
  • Lemma 3
  • ...and 3 more