From Inexact Gradients to Byzantine Robustness: Acceleration and Optimization under Similarity

TL;DR

Byzantine robustness in distributed optimization is reframed as optimization with inexact gradient oracles, enabling systematic use of first-order methods. The authors show that robust aggregation induces a $(\zeta^2,\alpha)$-inexact gradient with $\zeta^2=\nu G^2$ and $\alpha=\nu B^2$, and prove the reduction is tight with respect to known lower bounds. They introduce two strategies: a Nesterov-type accelerated method for inexact gradients and PIGS, a Prox/Similarity-based approach that leverages a proxy loss to precondition updates, both yielding improved communication efficiency and robust convergence. Theoretical guarantees coupled with MNIST-based experiments demonstrate faster optimization and resilience to adversarial clients while reducing the communication burden.

Abstract

Standard federated learning algorithms are vulnerable to adversarial nodes, a.k.a. Byzantine failures. To solve this issue, robust distributed learning algorithms have been developed, which typically replace parameter averaging by robust aggregations. While generic conditions on these aggregations exist to guarantee the convergence of (Stochastic) Gradient Descent (SGD), the analyses remain rather ad-hoc. This hinders the development of more complex robust algorithms, such as accelerated ones. In this work, we show that Byzantine-robust distributed optimization can, under standard generic assumptions, be cast as a general optimization with inexact gradient oracles (with both additive and multiplicative error terms), an active field of research. This allows for instance to directly show that GD on top of standard robust aggregation procedures obtains optimal asymptotic error in the Byzantine setting. Going further, we propose two optimization schemes to speed up the convergence. The first one is a Nesterov-type accelerated scheme whose proof directly derives from accelerated inexact gradient results applied to our formulation. The second one hinges on Optimization under Similarity, in which the server leverages an auxiliary loss function that approximates the global loss. Both approaches allow to drastically reduce the communication complexity compared to previous methods, as we show theoretically and empirically.

Figures (6)

• Figure 1: Distributed Logistic Regression, with $20$ honest and $1$ Byzantine client in the mildly heterogeneous setting ($\beta=5$), with a $l2$-regularization $\mu = 10^{-2}$. Train loss $\mathcal{L}_{\mathcal{H}}({\bm{x}}_k) - \mathcal{L}_{\mathcal{H}}^*$ under A Little Is Enough (ALIE) attack.
• Figure 2: Impact of the learning rate $\eta$ on PIGS, in a highly heterogeneous setting $(\beta=1)$ with $20$ honest clients and either with $1$ Byzantine performing IPM or ALIE attack, or without Byzantine client. $l2$-regularization with $\mu = 10^{-3}$.
• Figure 3: Impact of the learning rate $\eta$ on PIGS,i.i.d. setting with $20$ honest clients and either with $5$ Byzantine performing IPM or ALIE attack, or without Byzantine client. $l2$-regularization with $\mu = 10^{-3}$.
• Figure 4: Distributed Logistic Regression, with $20$ honest and $1$ Byzantine client in the mildly heterogeneous setting ($\beta=5$), with a $l2$-regularization $\lambda = 10^{-3}$. Test Accuracy under A Little Is Enough (ALIE) attack.
• Figure 5: Impact of the learning rate $\eta$ on PIGS, in a highly heterogeneous setting $(\beta=1)$ with $20$ honest clients and either with $1$ Byzantine performing IPM or ALIE attack, or without Byzantine client. Penalization $\mu = 10^{-3}$. Test Accuracy.

Theorems & Definitions (33)

• Definition 2.1
• Theorem 2.3: allouah2024robust
• Definition 2.4: $(f,\nu)$-robustness allouah2023fixing
• Lemma 2.5
• Definition 2.6: $(\zeta^2,\alpha)$-inexact oracle
• Proposition 2.7
• Remark 2.8
• Definition 3.1
• Proposition 3.2: devolder2014first Section 2.3
• Theorem 3.3: Fast Gradient Method w. Inexact Oracles