Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

dSTAR: Straggler Tolerant and Byzantine Resilient Distributed SGD

Jiahe Yan, Pratik Chaudhari, Leonard Kleinrock

TL;DR

Distributed SGD faces dual challenges from slow stragglers and Byzantine faults. The authors propose dSTAR, a light-weight method that waits for the fastest $k$ gradients and filters them using deviations from an ensemble median against a local validation gradient, achieving $(\alpha-f)$-Byzantine resilience and linear convergence. The approach offers $O(Nd)$ time complexity per iteration and tunable synchronization through $k$, demonstrated to outperform several GARs under diverse Byzantine attacks on Fashion-MNIST and CIFAR-10. Theoretical guarantees are complemented by empirical results showing robust accuracy across attacks and reduced iteration times, indicating practical applicability in adversarial or lag-prone distributed environments.

Abstract

Distributed model training needs to be adapted to challenges such as the straggler effect and Byzantine attacks. When coordinating the training process with multiple computing nodes, ensuring timely and reliable gradient aggregation amidst network and system malfunctions is essential. To tackle these issues, we propose \textit{dSTAR}, a lightweight and efficient approach for distributed stochastic gradient descent (SGD) that enhances robustness and convergence. \textit{dSTAR} selectively aggregates gradients by collecting updates from the first \(k\) workers to respond, filtering them based on deviations calculated using an ensemble median. This method not only mitigates the impact of stragglers but also fortifies the model against Byzantine adversaries. We theoretically establish that \textit{dSTAR} is (\(α, f\))-Byzantine resilient and achieves a linear convergence rate. Empirical evaluations across various scenarios demonstrate that \textit{dSTAR} consistently maintains high accuracy, outperforming other Byzantine-resilient methods that often suffer up to a 40-50\% accuracy drop under attack. Our results highlight \textit{dSTAR} as a robust solution for training models in distributed environments prone to both straggler delays and Byzantine faults.

dSTAR: Straggler Tolerant and Byzantine Resilient Distributed SGD

TL;DR

Distributed SGD faces dual challenges from slow stragglers and Byzantine faults. The authors propose dSTAR, a light-weight method that waits for the fastest gradients and filters them using deviations from an ensemble median against a local validation gradient, achieving -Byzantine resilience and linear convergence. The approach offers time complexity per iteration and tunable synchronization through , demonstrated to outperform several GARs under diverse Byzantine attacks on Fashion-MNIST and CIFAR-10. Theoretical guarantees are complemented by empirical results showing robust accuracy across attacks and reduced iteration times, indicating practical applicability in adversarial or lag-prone distributed environments.

Abstract

Distributed model training needs to be adapted to challenges such as the straggler effect and Byzantine attacks. When coordinating the training process with multiple computing nodes, ensuring timely and reliable gradient aggregation amidst network and system malfunctions is essential. To tackle these issues, we propose \textit{dSTAR}, a lightweight and efficient approach for distributed stochastic gradient descent (SGD) that enhances robustness and convergence. \textit{dSTAR} selectively aggregates gradients by collecting updates from the first workers to respond, filtering them based on deviations calculated using an ensemble median. This method not only mitigates the impact of stragglers but also fortifies the model against Byzantine adversaries. We theoretically establish that \textit{dSTAR} is ()-Byzantine resilient and achieves a linear convergence rate. Empirical evaluations across various scenarios demonstrate that \textit{dSTAR} consistently maintains high accuracy, outperforming other Byzantine-resilient methods that often suffer up to a 40-50\% accuracy drop under attack. Our results highlight \textit{dSTAR} as a robust solution for training models in distributed environments prone to both straggler delays and Byzantine faults.

Paper Structure

This paper contains 16 sections, 6 theorems, 31 equations, 4 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related work
  3. Contributions
  4. Assumptions
  5. Algorithm
  6. Time complexity
  7. Byzantine resilience analysis
  8. Convergence analysis
  9. Experiments
  10. Results
  11. Discussion and conclusion
  12. Appendix
  13. Byzantine resilience analysis
  14. Convergence analysis
  15. Full algorithm
  16. ...and 1 more sections

Key Result

Lemma 5.1

(Proof in the appendix) Under assumptions A1 to A3, if $g_*^t$ denotes the aggregated gradient for iteration $t$, it satisifies:

Figures (4)

  • Figure 1: Fashion-MNIST with "Empire" attack
  • Figure 2: Fashion-MNIST with "Little" attack
  • Figure 3: CIFAR10 with "Empire" attack
  • Figure 4: CIFAR10 with "Little" attack

Theorems & Definitions (9)

  • Definition 2.1: $(\alpha, f)$-Byzantine Resilience
  • Lemma 5.1
  • Lemma 5.2
  • Theorem 5.3
  • Remark 5.4
  • Theorem 5.5
  • Lemma A.1
  • Theorem A.2
  • proof