On the Tradeoff between Privacy Preservation and Byzantine-Robustness in Decentralized Learning
Haoxiang Ye, Heng Zhu, Qing Ling
TL;DR
This paper tackles the joint problem of privacy preservation and Byzantine-robustness in decentralized learning by proposing a generic SGD framework that injects Gaussian noise for differential privacy and employs robust aggregation rules to mitigate Byzantine attacks. It formalizes the impact of privacy noise on learning through a contraction-based analysis using a virtual mixing matrix $W$ and a contraction constant $\\rho$, revealing a fundamental tradeoff: higher noise improves privacy but worsens learning accuracy under Byzantine adversaries. The authors unify and quantify the mixing abilities of state-of-the-art robust rules (Trimmed Mean, SCC, IOS), showing that rules with smaller $\\rho$ and near-doubly stochastic $W$ yield more favorable privacy-robustness tradeoffs, with IOS often outperforming others in practice. Theoretical results are complemented by extensive experiments on MNIST and CIFAR10, including attacks and large networks, confirming the key insights and providing design guidelines for robust, privacy-preserving decentralized learning systems.
Abstract
This paper jointly considers privacy preservation and Byzantine-robustness in decentralized learning. In a decentralized network, honest-but-curious agents faithfully follow the prescribed algorithm, but expect to infer their neighbors' private data from messages received during the learning process, while dishonest-and-Byzantine agents disobey the prescribed algorithm, and deliberately disseminate wrong messages to their neighbors so as to bias the learning process. For this novel setting, we investigate a generic privacy-preserving and Byzantine-robust decentralized stochastic gradient descent (SGD) framework, in which Gaussian noise is injected to preserve privacy and robust aggregation rules are adopted to counteract Byzantine attacks. We analyze its learning error and privacy guarantee, discovering an essential tradeoff between privacy preservation and Byzantine-robustness in decentralized learning -- the learning error caused by defending against Byzantine attacks is exacerbated by the Gaussian noise added to preserve privacy. For a class of state-of-the-art robust aggregation rules, we give unified analysis of the "mixing abilities". Building upon this analysis, we reveal how the "mixing abilities" affect the tradeoff between privacy preservation and Byzantine-robustness. The theoretical results provide guidelines for achieving a favorable tradeoff with proper design of robust aggregation rules. Numerical experiments are conducted and corroborate our theoretical findings.