PFedDST: Personalized Federated Learning with Decentralized Selection Training

TL;DR

The paper tackles non-IID data and device heterogeneity in Federated Learning by introducing PFedDST, a decentralized personalized learning framework with strategic peer selection. Each client maintains a personalized header $h_i$ and a shared feature extractor $e_i$, aggregating $e_i$ from strategically chosen peers while keeping $h_i$ local, and training in two phases: first updating $e_i$ with frozen $h_i^f$, then updating $h_i$ with frozen $e_i^f$. Peer selection relies on a holistic score $\mathcal{S} = s_p(\alpha s_l - s_d + c)$ that combines loss disparity $s_l$, header distance cosine similarity $s_d$, and peer recency $s_p$, guiding efficient and diverse exchanges. Evaluations on CIFAR-10/100 with pathological partitions show PFedDST achieves faster convergence and higher personalized accuracy than centralized and decentralized baselines, demonstrating robustness to heterogeneity and improved communication efficiency through partial freezing and selective aggregation. These results suggest PFedDST is well-suited for real-world, resource-constrained, privacy-conscious distributed learning scenarios.

Abstract

Distributed Learning (DL) enables the training of machine learning models across multiple devices, yet it faces challenges like non-IID data distributions and device capability disparities, which can impede training efficiency. Communication bottlenecks further complicate traditional Federated Learning (FL) setups. To mitigate these issues, we introduce the Personalized Federated Learning with Decentralized Selection Training (PFedDST) framework. PFedDST enhances model training by allowing devices to strategically evaluate and select peers based on a comprehensive communication score. This score integrates loss, task similarity, and selection frequency, ensuring optimal peer connections. This selection strategy is tailored to increase local personalization and promote beneficial peer collaborations to strengthen the stability and efficiency of the training process. Our experiments demonstrate that PFedDST not only enhances model accuracy but also accelerates convergence. This approach outperforms state-of-the-art methods in handling data heterogeneity, delivering both faster and more effective training in diverse and decentralized systems.

Figures (4)

• Figure 1: Overview of PFedDST
• Figure 2: The validation result of each selected peer in the local data
• Figure 3: Test accuracy on CIFAR-10
• Figure 4: Test accuracy on CIFAR-100