Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FRAIN to Train: A Fast-and-Reliable Solution for Decentralized Federated Learning

Sanghyeon Park, Soo-Mook Moon

TL;DR

FRAIN tackles asynchronous decentralized federated learning under non-IID data, latency, and Byzantine threats by introducing FastSync to avoid full history replay and SLERP-based model merging to preserve update directions. It builds on BRAIN with a WiMA-like smoothing of recent medians and a staleness penalty to temper outdated proposals, achieving more stable convergence than FedAvg, FedAsync, and BRAIN in harsh environments. Empirical results on CNN/CIFAR-10 and a Transformer-based language model show enhanced robustness to delays and adversarial updates, with ablations confirming the efficacy of FastSync, SLERP, and staleness strategies. The approach offers practical benefits for decentralized FL by reducing bootstrapping costs, mitigating drift, and tolerating up to substantial Byzantine participation, albeit with static hyperparameters that suggest avenues for adaptive optimization.

Abstract

Federated learning (FL) enables collaborative model training across distributed clients while preserving data locality. Although FedAvg pioneered synchronous rounds for global model averaging, slower devices can delay collective progress. Asynchronous FL (e.g., FedAsync) addresses stragglers by continuously integrating client updates, yet naive implementations risk client drift due to non-IID data and stale contributions. Some Blockchain-based FL approaches (e.g., BRAIN) employ robust weighting or scoring of updates to resist malicious or misaligned proposals. However, performance drops can still persist under severe data heterogeneity or high staleness, and synchronization overhead has emerged as a new concern due to its aggregator-free architectures. We introduce Fast-and-Reliable AI Network, FRAIN, a new asynchronous FL method that mitigates these limitations by incorporating two key ideas. First, our FastSync strategy eliminates the need to replay past model versions, enabling newcomers and infrequent participants to efficiently approximate the global model. Second, we adopt spherical linear interpolation (SLERP) when merging parameters, preserving models' directions and alleviating destructive interference from divergent local training. Experiments with a CNN image-classification model and a Transformer-based language model demonstrate that FRAIN achieves more stable and robust convergence than FedAvg, FedAsync, and BRAIN, especially under harsh environments: non-IID data distributions, networks that experience delays and require frequent re-synchronization, and the presence of malicious nodes.

FRAIN to Train: A Fast-and-Reliable Solution for Decentralized Federated Learning

TL;DR

FRAIN tackles asynchronous decentralized federated learning under non-IID data, latency, and Byzantine threats by introducing FastSync to avoid full history replay and SLERP-based model merging to preserve update directions. It builds on BRAIN with a WiMA-like smoothing of recent medians and a staleness penalty to temper outdated proposals, achieving more stable convergence than FedAvg, FedAsync, and BRAIN in harsh environments. Empirical results on CNN/CIFAR-10 and a Transformer-based language model show enhanced robustness to delays and adversarial updates, with ablations confirming the efficacy of FastSync, SLERP, and staleness strategies. The approach offers practical benefits for decentralized FL by reducing bootstrapping costs, mitigating drift, and tolerating up to substantial Byzantine participation, albeit with static hyperparameters that suggest avenues for adaptive optimization.

Abstract

Federated learning (FL) enables collaborative model training across distributed clients while preserving data locality. Although FedAvg pioneered synchronous rounds for global model averaging, slower devices can delay collective progress. Asynchronous FL (e.g., FedAsync) addresses stragglers by continuously integrating client updates, yet naive implementations risk client drift due to non-IID data and stale contributions. Some Blockchain-based FL approaches (e.g., BRAIN) employ robust weighting or scoring of updates to resist malicious or misaligned proposals. However, performance drops can still persist under severe data heterogeneity or high staleness, and synchronization overhead has emerged as a new concern due to its aggregator-free architectures. We introduce Fast-and-Reliable AI Network, FRAIN, a new asynchronous FL method that mitigates these limitations by incorporating two key ideas. First, our FastSync strategy eliminates the need to replay past model versions, enabling newcomers and infrequent participants to efficiently approximate the global model. Second, we adopt spherical linear interpolation (SLERP) when merging parameters, preserving models' directions and alleviating destructive interference from divergent local training. Experiments with a CNN image-classification model and a Transformer-based language model demonstrate that FRAIN achieves more stable and robust convergence than FedAvg, FedAsync, and BRAIN, especially under harsh environments: non-IID data distributions, networks that experience delays and require frequent re-synchronization, and the presence of malicious nodes.
Paper Structure (30 sections, 1 theorem, 18 equations, 7 figures, 1 table, 1 algorithm)

This paper contains 30 sections, 1 theorem, 18 equations, 7 figures, 1 table, 1 algorithm.

Table of Contents

  1. Background
  2. Federated Learning
  3. FedAvg.
  4. FedAsync.
  5. BRAIN
  6. Overview.
  7. Aggregator-Free Synchronization.
  8. Window-based Model Averaging
  9. Spherical Linear Interpolation
  10. FRAIN
  11. Overview
  12. FastSync.
  13. Model Merging.
  14. Fast Synchronization
  15. SLERP-Based Model Merging
  16. ...and 15 more sections

Key Result

Theorem 1

Let $\overline{M_r}^{(\textsc{BRAIN})}$ and $\overline{M_r}^{(\textsc{WiMA})}$ be global model sequences, both initialized to the same $\overline{M_0}$ yet updated with different mixing weights: where $a_r \in [\mathcal{T},\,1]$ ($0<\mathcal{T}<1$), and $N\ge1$ is an integer window size. At round $r$, each method $m \in \{\textsc{WiMA}, \textsc{BRAIN}\}$ updates its global model as: We assume th

Figures (7)

  • Figure 1: Process flow for both FastSync (dotted green) and Slerp-based model merging (orange) in FRAIN.
  • Figure 2: Comparison between LERP and SLERP.
  • Figure 3: Accuracy ($\uparrow$) across gradient updates on CNN/CIFAR-10.
  • Figure 4: PPL ($\downarrow$) across gradient updates on sLM-135M/WikiText-2.
  • Figure 5: Gradient updates vs. (a) CNN/CIFAR-10 accuracy with nullifiers, and (b) sLM-135M/WikiText-2 perplexity with randomizers. Both were simulated under a non‑IID environment.
  • ...and 2 more figures

Theorems & Definitions (2)

  • Theorem 1: Bound on WiMA--BRAIN Global Model Difference
  • proof : Proof (Sketch)