Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Divergence-Based Adaptive Aggregation for Byzantine Robust Federated Learning

Bingnan Xiao, Feng Zhu, Jingjing Zhang, Wei Ni, Xin Wang

TL;DR

DRAG introduces a divergence-based adaptive aggregation that uses a global reference direction $\mathbf{r}^t$ to align local updates with the global descent direction, mitigating client drift in non-IID FL; BR-DRAG extends DRAG with a vetted root dataset to withstand Byzantine attacks, with provable convergence for non-convex objectives even with arbitrary attack proportions. Theoretical results bound the average gradient norm by a term that scales with $O\left(\frac{1}{\eta U T}\right)$ plus a non-vanishing variance term, and experiments on EMNIST, CIFAR-10, CIFAR-100 show faster convergence and robustness against attacks compared to state-of-the-art baselines. The combination of divergence-based gradient modification and root-data-driven reference directions provides a practical, communication-efficient approach to Byzantine-robust federated learning with heterogeneous data. The work suggests future extensions to decentralized FL and more scalable defense strategies while preserving convergence guarantees under adversarial conditions.

Abstract

Inherent client drifts caused by data heterogeneity, as well as vulnerability to Byzantine attacks within the system, hinder effective model training and convergence in federated learning (FL). This paper presents two new frameworks, named DiveRgence-based Adaptive aGgregation (DRAG) and Byzantine-Resilient DRAG (BR-DRAG), to mitigate client drifts and resist attacks while expediting training. DRAG designs a reference direction and a metric named divergence of degree to quantify the deviation of local updates. Accordingly, each worker can align its local update via linear calibration without extra communication cost. BR-DRAG refines DRAG under Byzantine attacks by maintaining a vetted root dataset at the server to produce trusted reference directions. The workers' updates can be then calibrated to mitigate divergence caused by malicious attacks. We analytically prove that DRAG and BR-DRAG achieve fast convergence for non-convex models under partial worker participation, data heterogeneity, and Byzantine attacks. Experiments validate the effectiveness of DRAG and its superior performance over state-of-the-art methods in handling client drifts, and highlight the robustness of BR-DRAG in maintaining resilience against data heterogeneity and diverse Byzantine attacks.

Divergence-Based Adaptive Aggregation for Byzantine Robust Federated Learning

TL;DR

DRAG introduces a divergence-based adaptive aggregation that uses a global reference direction to align local updates with the global descent direction, mitigating client drift in non-IID FL; BR-DRAG extends DRAG with a vetted root dataset to withstand Byzantine attacks, with provable convergence for non-convex objectives even with arbitrary attack proportions. Theoretical results bound the average gradient norm by a term that scales with plus a non-vanishing variance term, and experiments on EMNIST, CIFAR-10, CIFAR-100 show faster convergence and robustness against attacks compared to state-of-the-art baselines. The combination of divergence-based gradient modification and root-data-driven reference directions provides a practical, communication-efficient approach to Byzantine-robust federated learning with heterogeneous data. The work suggests future extensions to decentralized FL and more scalable defense strategies while preserving convergence guarantees under adversarial conditions.

Abstract

Inherent client drifts caused by data heterogeneity, as well as vulnerability to Byzantine attacks within the system, hinder effective model training and convergence in federated learning (FL). This paper presents two new frameworks, named DiveRgence-based Adaptive aGgregation (DRAG) and Byzantine-Resilient DRAG (BR-DRAG), to mitigate client drifts and resist attacks while expediting training. DRAG designs a reference direction and a metric named divergence of degree to quantify the deviation of local updates. Accordingly, each worker can align its local update via linear calibration without extra communication cost. BR-DRAG refines DRAG under Byzantine attacks by maintaining a vetted root dataset at the server to produce trusted reference directions. The workers' updates can be then calibrated to mitigate divergence caused by malicious attacks. We analytically prove that DRAG and BR-DRAG achieve fast convergence for non-convex models under partial worker participation, data heterogeneity, and Byzantine attacks. Experiments validate the effectiveness of DRAG and its superior performance over state-of-the-art methods in handling client drifts, and highlight the robustness of BR-DRAG in maintaining resilience against data heterogeneity and diverse Byzantine attacks.
Paper Structure (21 sections, 2 theorems, 46 equations, 17 figures, 2 algorithms)

This paper contains 21 sections, 2 theorems, 46 equations, 17 figures, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Contribution
  4. System Model
  5. Federated Learning Framework
  6. Byzantine Attack
  7. DiveRgence-based Adaptive aGgregation
  8. Algorithm Overview
  9. Global Reference Direction
  10. Divergence-based Local Gradient Modification
  11. DiveRgence-based Adaptive aGgregation
  12. Algorithm Overview
  13. Byzantine-Resilient Global Reference Direction
  14. Byzantine-Resilient Local Gradient Modification
  15. Convergence Analysis of DRAG and BR-DRAG
  16. ...and 6 more sections

Key Result

Theorem 1

Suppose that $F_m, \forall m \in \mathcal{M}$ is non-convex, and the stepsize is $\eta\leq \frac{1}{8LU}$. Given an initial model $\boldsymbol{\theta}^0$, after $T$ training rounds, DRAG guarantees where $V=\frac{1}{\gamma\eta U} ( c \eta (2\sigma_L^2+9 \sigma_G^2)+\frac{\eta^2 U^2 L}{2}\left(\sigma_L^2+3 U \sigma_G^2\right) + (\frac{\eta^3 U^2 L^2(85 c+5)}{2}+\frac{15 \eta^4 U^3 L^3}{2})(\sigma

Figures (17)

  • Figure 1: The architecture of a FL system with $S$ selected workers in round $t$. For Byzantine FL systems, the attacked workers can upload arbitrary local updates to affect global model aggregation.
  • Figure 2: Illustration of vector modification of DRAG. For workers with different DoD values, the modified gradient $\mathbf{v}_m^t$ (solid black line) has a larger component on the reference direction $\mathbf{r}^{t}$ (solid yellow line) than the raw local update $\mathbf{g}_{m}^t$ (solid blue line).
  • Figure 3: Convergence performance of different algorithms on EMNIST.
  • Figure 4: Convergence performance of different algorithms on CIFAR-10.
  • Figure 5: Convergence performance of different algorithms on CIFAR-100.
  • ...and 12 more figures

Theorems & Definitions (4)

  • Theorem 1
  • proof
  • Theorem 2
  • proof