Topology-Independent Robustness of the Weighted Mean under Label Poisoning Attacks in Heterogeneous Decentralized Learning

TL;DR

This work analyzes resilience to label poisoning in heterogeneous decentralized learning, contrasting robust aggregators with the simple weighted mean. It shows that robust aggregators incur topology-dependent learning errors, whereas the weighted mean achieves topology-independent performance under label poisoning, and can even outperform robust methods when data heterogeneity is high. The authors provide tight upper and lower bounds, demonstrate their results both theoretically and empirically, and identify concrete network-topology scenarios where the weighted mean is advantageous. The findings offer practical guidance for aggregator choice in distributed systems with non-i.i.d. data and adversarial agents, and highlight topology as a central factor in robustness gains.

Abstract

Robustness to malicious attacks is crucial for practical decentralized signal processing and machine learning systems. A typical example of such attacks is label poisoning, meaning that some agents possess corrupted local labels and share models trained on these poisoned data. To defend against malicious attacks, existing works often focus on designing robust aggregators; meanwhile, the weighted mean aggregator is typically considered a simple, vulnerable baseline. This paper analyzes the robustness of decentralized gradient descent under label poisoning attacks, considering both robust and weighted mean aggregators. Theoretical results reveal that the learning errors of robust aggregators depend on the network topology, whereas the performance of weighted mean aggregator is topology-independent. Remarkably, the weighted mean aggregator, although often considered vulnerable, can outperform robust aggregators under sufficient heterogeneity, particularly when: (i) the global contamination rate (i.e., the fraction of poisoned agents for the entire network) is smaller than the local contamination rate (i.e., the maximal fraction of poisoned neighbors for the regular agents); (ii) the network of regular agents is disconnected; or (iii) the network of regular agents is sparse and the local contamination rate is high. Empirical results support our theoretical findings, highlighting the important role of network topology in the robustness to label poisoning attacks.

Figures (13)

• Figure 1: Classification accuracies of the softmax regression model trained on the non-i.i.d. MNIST dataset under label flipping attacks for: (a) the two-castle graph, (b) the fan graph. The blue and red points represent the regular and poisoned agents, respectively. The classification accuracy is in terms of the average model of all regular agents. WeiMean stands for the weighted mean, with the weights constructed using the Metropolis-Hastings rule he2022byzantine.
• Figure 2: Classification accuracies and consensus errors of softmax regression model trained on MNIST in the two-castle graph.
• Figure 3: Classification accuracies and consensus errors of ResNet18 trained on CIFAR100 in the two-castle graph.
• Figure 4: Heterogeneity of regular local gradients and disturbances of poisoned local gradients for softmax regression trained on MNIST and ResNet18 trained on CIFAR100 in the two-castle graph.
• Figure 5: Classification accuracies and consensus errors of softmax regression trained on MNIST in the line graph.

Theorems & Definitions (20)

• Definition 1: $(\rho, M)$-robust aggregator
• Theorem 1
• Theorem 2
• Remark 1
• Definition 2: Majority-dominant aggregator
• Theorem 3
• Remark 2
• proof
• proof
• proof