VALID: a Validated Algorithm for Learning in Decentralized Networks with Possible Adversarial Presence

TL;DR

The paper introduces VALID, a validated learning protocol for decentralized networks facing potential Byzantine agents and heterogeneous data. By combining a learning phase of distributed SGD with a rigorous validation phase leveraging validated broadcast and polynomial hashing, VALID either achieves an admissible consensus or flags adversaries, while maintaining an $O(1/T)$ convergence rate and comparable complexity to non-adversarial SGD. A heterogeneity-based optimality test and a formal admissible consensus framework underpin the detection guarantees, enabling fast performance without sacrificing robustness in adversary-free settings. Empirical results on a 20-node network with MNIST data show that VALID outperforms other Byzantine-robust methods when no attacks are present and reliably detects adversaries when attacks occur, highlighting practical impact for secure, scalable decentralized learning.

Abstract

We introduce the paradigm of validated decentralized learning for undirected networks with heterogeneous data and possible adversarial infiltration. We require (a) convergence to a global empirical loss minimizer when adversaries are absent, and (b) either detection of adversarial presence of convergence to an admissible consensus irrespective of the adversarial configuration. To this end, we propose the VALID protocol which, to the best of our knowledge, is the first to achieve a validated learning guarantee. Moreover, VALID offers an O(1/T) convergence rate (under pertinent regularity assumptions), and computational and communication complexities comparable to non-adversarial distributed stochastic gradient descent. Remarkably, VALID retains optimal performance metrics in adversary-free environments, sidestepping the robustness penalties observed in prior byzantine-robust methods. A distinctive aspect of our study is a heterogeneity metric based on the norms of individual agents' gradients computed at the global empirical loss minimizer. This not only provides a natural statistic for detecting significant byzantine disruptions but also allows us to prove the optimality of VALID in wide generality. Lastly, our numerical results reveal that, in the absence of adversaries, VALID converges faster than state-of-the-art byzantine robust algorithms, while when adversaries are present, VALID terminates with each honest either converging to an admissible consensus of declaring adversarial presence in the network.

Figures (8)

• Figure 1: This figure depicts the $t$-th round of a validated learning protocol (for $t=1,2,\ldots, T$). Agent $v$ is an honest agent and agent $c$ is a Byzantine agent. The rectangles next to agents $v$ and agent $c$ denote the available information to them at the beginning of the $t$-th round.
• Figure 2: At the conclusion of the $T$-th round, agent $v$ either sets its validation state $S_v$ to $\boldsymbol{\top}$ to indicate a valid consensus or to $\boldsymbol{\perp}$ to declare Byzantine presence.
• Figure 3: The above figure shows the messages passed to node $v$ by its neighbors in round $t-1$ and the messages passed from node $v$ to its neighbor in round $t$ of the LearnModel protocol. Our LocalValidation protocol verifies if these messages are consistent with each other. For instance, if agent $v$ performs all computations honestly, the messages for a single round must satisfy the property that $\vecx_{va}^{(t)}= \vecx_{vb}^{(t)}=\vecx_{vc}^{(t)}$ and $\vecg_{va}^{(t)}= \vecg_{vb}^{(t)}=\vecg_{vc}^{(t)}$ for all $t$. Further, the messages passed in different rounds are related by the equality $\vecx_{va}^{(t)}=(1-3\eta^{(t)}) \vecx_{va}^{(t-1)}+\eta^{(t)}(\vecx_{av}^{(t-1)}+\vecx_{bv}^{(t-1)}+\vecx_{cv}^{(t-1)})-\vecg_{va}^{(t)}$.
• Figure 4: Performance of Valid compared with UBAR and Bridge-M under non-i.i.d. setting.
• Figure 5: Performance of Valid , UBAR, and Bridge-M under Gaussian attack (1 adversarial agent, non-i.i.d. setting).

Theorems & Definitions (13)

• Definition 1: $\delta$-Heterogeneous Distributions
• Remark 1
• Definition 2: $(\cU, P_{\cV}, \delta)$-Admissible Consensus Models
• Theorem 1: The Valid protocol
• Remark 2
• Theorem 2: Optimality of Valid
• Theorem 3: JakovBSK:18SahuJBK:18
• Lemma 4: Bounded iterates
• proof
• Lemma 5: Validated models