Robust and Efficient Collaborative Learning

TL;DR

This work tackles robust decentralized collaborative learning in the presence of Byzantine adversaries without relying on a central server. It introduces Robust Pull-based Epidemic Learning (RPEL), a pull-based epidemic communication scheme that achieves $O(n \log n)$ communication by pulling model updates from a small random subset of peers. The authors define the Effective adversarial fraction and provide convergence guarantees for non-convex objectives under mild assumptions, using robust aggregation and variance reduction, with high-probability robustness against omniscient attacks. Empirically, RPEL achieves competitive accuracy on MNIST and CIFAR-10 with up to 20% adversaries, while significantly reducing communication compared to all-to-all robust methods, and it scales effectively to large networks. This work thus offers a practical, scalable, serverless approach to robust distributed learning with strong theoretical and empirical support.

Abstract

Collaborative machine learning is challenged by training-time adversarial behaviors. Existing approaches to tolerate such behaviors either rely on a central server or induce high communication costs. We propose Robust Pull-based Epidemic Learning (RPEL), a novel, scalable collaborative approach to ensure robust learning despite adversaries. RPEL does not rely on any central server and, unlike traditional methods, where communication costs grow in $\mathcal{O}(n^2)$ with the number of nodes $n$, RPEL employs a pull-based epidemic-based communication strategy that scales in $\mathcal{O}(n \log n)$. By pulling model parameters from small random subsets of nodes, RPEL significantly lowers the number of required messages without compromising convergence guarantees, which hold with high probability. Empirical results demonstrate that RPEL maintains robustness in adversarial settings, competes with all-to-all communication accuracy, and scales efficiently across large networks.

Figures (21)

• Figure 1: Test accuracies obtained on MNIST. (Left) $n=100, b=10, s=15$ , (right) $n=30, b=6, s=15$.
• Figure 2: Test accuracies obtained on CIFAR-10 with $n=20, b=3$ (Left) $s=6$, (Right) $s=19$.
• Figure 3: Effective adversarial fraction simulation. For equal adversarial fraction, increasing the number of participants does not require increasing the number of selected neighbors by much.
• Figure 4: Average test accuracy on MNIST - ALIE attack
• Figure 5: Worst test accuracy on MNIST - ALIE attack

Theorems & Definitions (24)

• Definition 3.1: Byzantine Resilience
• Lemma 4.0
• Definition 5.1: $(s,\hat{b}, \kappa)$-robustness allouah2023fixingmixingrecipeoptimal
• Lemma 5.1
• Theorem 5.5: Convergence of \ref{['alg:BEL']}
• Corollary 5.5
• Lemma A.1
• proof
• Lemma A.2
• proof