Efficient Node Selection in Private Personalized Decentralized Learning

TL;DR

The paper addresses privacy risks in private personalized decentralized learning by proposing PPDDL, which preserves data privacy by observing only aggregated model updates through secure aggregation and by guiding collaborator selection with a correlated adversarial multi-armed bandit. It defines a scalable group-selection process where each node evaluates M-sized neighbor groups, uses pseudo-rewards to exploit overlaps between groups, and applies Tsallis-Inf with a non-increasing q(t) to balance exploration and exploitation. The number of arms is Ci, equal to the number of M-sized neighbor groups, and pseudo-rewards s_{l,j}^{(t)}(r_j^{(t)}) depend on overlap u_{l,j} and time, enabling efficient learning under privacy constraints. Empirical results on CIFAR-10 and Fashion-MNIST under covariate and label shift show PPDDL achieves competitive accuracy with privacy, outperforming or matching non-private baselines like DAC in several settings, and illustrating the method’s practicality for privacy-preserving decentralized learning.

Abstract

Personalized decentralized learning is a promising paradigm for distributed learning, enabling each node to train a local model on its own data and collaborate with other nodes to improve without sharing any data. However, this approach poses significant privacy risks, as nodes may inadvertently disclose sensitive information about their data or preferences through their collaboration choices. In this paper, we propose Private Personalized Decentralized Learning (PPDL), a novel approach that combines secure aggregation and correlated adversarial multi-armed bandit optimization to protect node privacy while facilitating efficient node selection. By leveraging dependencies between different arms, represented by potential collaborators, we demonstrate that PPDL can effectively identify suitable collaborators solely based on aggregated models. Additionally, we show that PPDL surpasses previous non-private methods in model performance on standard benchmarks under label and covariate shift scenarios.

Figures (1)

• Figure 1: Heatmaps visualising how often node $x$ communicates with node $y$ for the four different methods on the CIFAR-10 dataset with 5 clusters.