Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Personalized Federated Learning with Attention-based Client Selection

Zihan Chen, Jundong Li, Cong Shen

TL;DR

Non-IID data across clients challenges traditional federated learning and is exacerbated by data scarcity. FedACS introduces an attention-based client selection mechanism within a personalized federated learning framework, optimizing a joint objective $F_\lambda(W)=F(W)+\lambda R(W)$ with $F(W)=\sum_i F_i(w_i)$ and $R(W)=\sum_{i,j} s_{ij}\|w_i-w_j\|^2$, where $s_{ij}$ are cosine similarities; optimization proceeds incrementally via intermediate models and a dynamic threshold $\delta^k$. The authors prove convergence to a first-order stationary point under standard assumptions and demonstrate an $\mathcal{O}(1/\sqrt{K})$ rate, complemented by empirical validation on CIFAR-10 and FMNIST that shows improved personalization and robustness under non-IID data and limited data. Overall, FedACS offers a practical, theoretically grounded approach to personalized federated learning with data heterogeneity, delivering tangible gains in real-world settings.

Abstract

Personalized Federated Learning (PFL) relies on collective data knowledge to build customized models. However, non-IID data between clients poses significant challenges, as collaborating with clients who have diverse data distributions can harm local model performance, especially with limited training data. To address this issue, we propose FedACS, a new PFL algorithm with an Attention-based Client Selection mechanism. FedACS integrates an attention mechanism to enhance collaboration among clients with similar data distributions and mitigate the data scarcity issue. It prioritizes and allocates resources based on data similarity. We further establish the theoretical convergence behavior of FedACS. Experiments on CIFAR10 and FMNIST validate FedACS's superiority, showcasing its potential to advance personalized federated learning. By tackling non-IID data challenges and data scarcity, FedACS offers promising advances in the field of personalized federated learning.

Personalized Federated Learning with Attention-based Client Selection

TL;DR

Non-IID data across clients challenges traditional federated learning and is exacerbated by data scarcity. FedACS introduces an attention-based client selection mechanism within a personalized federated learning framework, optimizing a joint objective with and , where are cosine similarities; optimization proceeds incrementally via intermediate models and a dynamic threshold . The authors prove convergence to a first-order stationary point under standard assumptions and demonstrate an rate, complemented by empirical validation on CIFAR-10 and FMNIST that shows improved personalization and robustness under non-IID data and limited data. Overall, FedACS offers a practical, theoretically grounded approach to personalized federated learning with data heterogeneity, delivering tangible gains in real-world settings.

Abstract

Personalized Federated Learning (PFL) relies on collective data knowledge to build customized models. However, non-IID data between clients poses significant challenges, as collaborating with clients who have diverse data distributions can harm local model performance, especially with limited training data. To address this issue, we propose FedACS, a new PFL algorithm with an Attention-based Client Selection mechanism. FedACS integrates an attention mechanism to enhance collaboration among clients with similar data distributions and mitigate the data scarcity issue. It prioritizes and allocates resources based on data similarity. We further establish the theoretical convergence behavior of FedACS. Experiments on CIFAR10 and FMNIST validate FedACS's superiority, showcasing its potential to advance personalized federated learning. By tackling non-IID data challenges and data scarcity, FedACS offers promising advances in the field of personalized federated learning.
Paper Structure (14 sections, 2 theorems, 32 equations, 1 figure, 2 tables, 1 algorithm)

This paper contains 14 sections, 2 theorems, 32 equations, 1 figure, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Personalized Federated Learning
  3. Formulation
  4. Our proposed method: FedACS
  5. Relation to FedAMP
  6. Convergence analysis of FedACS
  7. Experiments
  8. Experimental settings
  9. Results on the sufficient non-IID data setting
  10. Results on the insufficient non-IID data setting
  11. Ablation study of data distribution
  12. Conclusion
  13. Missing Proofs
  14. Proof of Theorem \ref{['theory:main']}

Key Result

Theorem 2.1

Assuming the validity of Assumption 1 and considering the continuous differentiability of functions $\mathcal{R}(W)$ and $\mathcal{F}(W)$, with gradients that exhibit Lipschitz continuity with a modulus $L$, if $\beta_1=\beta_2=\dots=\beta_K=1\sqrt{K}$ and $\alpha_1=\alpha_2=\dots=\alpha_K=\lambda/\ where $\mathcal{F}_{\lambda}^{*} = \mathop{\mathrm{arg\,min}}\limits_{W}\mathcal{F}_{\lambda}^{W}$.

Figures (1)

  • Figure 1: Experiment results

Theorems & Definitions (3)

  • Theorem 2.1
  • Theorem A.1
  • proof