Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Resilient Peer-to-peer Learning based on Adaptive Aggregation

Chandreyee Bhowmick, Xenofon Koutsoukos

TL;DR

This work tackles resilient decentralized P2P learning in the presence of Byzantine workers, non-iid data, and non-convex losses. It introduces a loss-based adaptive aggregation that assigns neighbor weights based on the inverse of each neighbor's local risk $r_k(\hat{w}_l^t)$, computed from the neighbor's parameters and private data, with a variant that uses only neighbors with lower risk to enhance robustness. The authors prove convergence guarantees under local strong convexity and Lipschitz gradients, yielding a bounded optimality gap for fixed stepsize $\mu$ and aggregation complexity $O(d|\mathcal{N}_k|)$. Empirical evaluations on HAR, MNIST, and Spambase under several attack models show improved accuracy over baselines, demonstrating robust performance in non-iid, non-convex settings and highlighting the method's practicality for decentralized ML without a central server.

Abstract

Collaborative learning in peer-to-peer networks offers the benefits of distributed learning while mitigating the risks associated with single points of failure inherent in centralized servers. However, adversarial workers pose potential threats by attempting to inject malicious information into the network. Thus, ensuring the resilience of peer-to-peer learning emerges as a pivotal research objective. The challenge is exacerbated in the presence of non-convex loss functions and non-iid data distributions. This paper introduces a resilient aggregation technique tailored for such scenarios, aimed at fostering similarity among peers' learning processes. The aggregation weights are determined through an optimization procedure, and use the loss function computed using the neighbor's models and individual private data, thereby addressing concerns regarding data privacy in distributed machine learning. Theoretical analysis demonstrates convergence of parameters with non-convex loss functions and non-iid data distributions. Empirical evaluations across three distinct machine learning tasks support the claims. The empirical findings, which encompass a range of diverse attack models, also demonstrate improved accuracy when compared to existing methodologies.

Resilient Peer-to-peer Learning based on Adaptive Aggregation

TL;DR

This work tackles resilient decentralized P2P learning in the presence of Byzantine workers, non-iid data, and non-convex losses. It introduces a loss-based adaptive aggregation that assigns neighbor weights based on the inverse of each neighbor's local risk , computed from the neighbor's parameters and private data, with a variant that uses only neighbors with lower risk to enhance robustness. The authors prove convergence guarantees under local strong convexity and Lipschitz gradients, yielding a bounded optimality gap for fixed stepsize and aggregation complexity . Empirical evaluations on HAR, MNIST, and Spambase under several attack models show improved accuracy over baselines, demonstrating robust performance in non-iid, non-convex settings and highlighting the method's practicality for decentralized ML without a central server.

Abstract

Collaborative learning in peer-to-peer networks offers the benefits of distributed learning while mitigating the risks associated with single points of failure inherent in centralized servers. However, adversarial workers pose potential threats by attempting to inject malicious information into the network. Thus, ensuring the resilience of peer-to-peer learning emerges as a pivotal research objective. The challenge is exacerbated in the presence of non-convex loss functions and non-iid data distributions. This paper introduces a resilient aggregation technique tailored for such scenarios, aimed at fostering similarity among peers' learning processes. The aggregation weights are determined through an optimization procedure, and use the loss function computed using the neighbor's models and individual private data, thereby addressing concerns regarding data privacy in distributed machine learning. Theoretical analysis demonstrates convergence of parameters with non-convex loss functions and non-iid data distributions. Empirical evaluations across three distinct machine learning tasks support the claims. The empirical findings, which encompass a range of diverse attack models, also demonstrate improved accuracy when compared to existing methodologies.
Paper Structure (11 sections, 2 theorems, 8 equations, 3 figures, 1 algorithm)

This paper contains 11 sections, 2 theorems, 8 equations, 3 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries of Distributed Peer-to-peer Learning
  3. Background on Peer-to-peer Learning
  4. Attacks on Peer-to-peer Learning
  5. Peer-to-peer Learning using Loss-based Adaptive Aggregation
  6. Theoretical Analysis
  7. Evaluation
  8. Tasks and Datasets
  9. Attack Models and Baselines
  10. Results and Discussion
  11. Conclusion

Key Result

lemma 1

All proofs are included in the supplementary material. Suppose every worker be initialized with parameters within $\mathbb{B}(w_s^*, \Gamma)$. For every normal worker $k \in \mathcal{V}_c$ that uses aggregation step eq:aggregation_step with aggregation weights given by eq:aggregation_weights, the fo

Figures (3)

  • Figure 1: Test accuracy for activity recognition task with 13 adversarial workers.
  • Figure 2: Test accuracy for digit classification task with 3 adversarial workers.
  • Figure 3: Test accuracy for spam detection task with 3 adversarial workers.

Theorems & Definitions (3)

  • remark 1
  • lemma 1
  • theorem 1