Smart Sampling: Helping from Friendly Neighbors for Decentralized Federated Learning

TL;DR

This work tackles improving performance in Decentralized Federated Learning by proposing AFIND+, an adaptive neighbor collaboration framework. AFIND+ identifies helpful neighbors using a feature-proxy similarity, dynamically adjusts the number of cooperating neighbors with an adaptive threshold, and aggregates sampled models with contribution-aware Boltzmann weights, backed by a convergence guarantee for nonconvex objectives. Empirically, AFIND+ yields consistent accuracy gains (up to about 5% on CIFAR-10 and CIFAR-100 under diverse non-IID partitions) and faster convergence across FEMNIST, CIFAR-10, and CIFAR-100, while remaining compatible with various personalized FL techniques. These results suggest that similarity-based neighbor selection and contribution-weighted aggregation can substantially enhance decentralized collaboration in heterogeneous data environments.

Abstract

Federated Learning (FL) is gaining widespread interest for its ability to share knowledge while preserving privacy and reducing communication costs. Unlike Centralized FL, Decentralized FL (DFL) employs a network architecture that eliminates the need for a central server, allowing direct communication among clients and leading to significant communication resource savings. However, due to data heterogeneity, not all neighboring nodes contribute to enhancing the local client's model performance. In this work, we introduce \textbf{\emph{AFIND+}}, a simple yet efficient algorithm for sampling and aggregating neighbors in DFL, with the aim of leveraging collaboration to improve clients' model performance. AFIND+ identifies helpful neighbors, adaptively adjusts the number of selected neighbors, and strategically aggregates the sampled neighbors' models based on their contributions. Numerical results on real-world datasets with diverse data partitions demonstrate that AFIND+ outperforms other sampling algorithms in DFL and is compatible with most existing DFL optimization algorithms.

Figures (8)

• Figure 1: Illustration of Centralized FL and Decentralized FL. In centralized FL, communication takes place between the server and the clients, whereas in decentralized FL, it occurs directly between clients without the need for a central server.
• Figure 2: Toy examples show the importance of appropriate cooperation. Clients 1 and 2 have FashionMNIST datasets with labels {4, 5, 6, 7}, while Client 3's dataset has labels {0, 1, 2, 3}. Using Client 2 as a baseline, accuracy comparisons indicate that collaboration between clients with similar data (Clients 1 and 2) is beneficial, whereas cooperation between clients with diverse data (Clients 2 and 3) is detrimental.
• Figure 3: Illustration of the AFIND+ method. The node represents the clients in the network, with different sizes and colors indicating variations in data distribution. First, the client identifies helpful neighbors with similar data distribution, represented by a similar feature proxy. Next, it filters neighbors using a threshold based on overall distribution and helpfulness. Finally, it aggregates the selected clients based on their contributions.
• Figure 4: Comparison of convergence performance of DFL using different collaboration strategies. AFIND refers to our method AFIND+ with uniform aggregation.
• Figure 5: Improving the performance of Decentralized PFL methods compatible with AFIND+.

Theorems & Definitions (14)

• Definition 3.1: Coreset
• Definition 3.2: Coreset of DFL
• Remark 3.3
• Remark 3.4
• Theorem 4.1
• Proposition B.1
• Lemma B.2
• proof
• Lemma B.3
• proof