Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Bayesian Framework for Clustered Federated Learning

Peng Wu, Tales Imbiriba, Pau Closas

TL;DR

This work presents a unified Bayesian framework for clustered FL which associates clients to clusters and proposes several practical algorithms to handle the, otherwise growing, data associations in a way that trades off performance and computational complexity.

Abstract

One of the main challenges of federated learning (FL) is handling non-independent and identically distributed (non-IID) client data, which may occur in practice due to unbalanced datasets and use of different data sources across clients. Knowledge sharing and model personalization are key strategies for addressing this issue. Clustered federated learning is a class of FL methods that groups clients that observe similarly distributed data into clusters, such that every client is typically associated with one data distribution and participates in training a model for that distribution along their cluster peers. In this paper, we present a unified Bayesian framework for clustered FL which associates clients to clusters. Then we propose several practical algorithms to handle the, otherwise growing, data associations in a way that trades off performance and computational complexity. This work provides insights on client-cluster associations and enables client knowledge sharing in new ways. The proposed framework circumvents the need for unique client-cluster associations, which is seen to increase the performance of the resulting models in a variety of experiments.

A Bayesian Framework for Clustered Federated Learning

TL;DR

This work presents a unified Bayesian framework for clustered FL which associates clients to clusters and proposes several practical algorithms to handle the, otherwise growing, data associations in a way that trades off performance and computational complexity.

Abstract

One of the main challenges of federated learning (FL) is handling non-independent and identically distributed (non-IID) client data, which may occur in practice due to unbalanced datasets and use of different data sources across clients. Knowledge sharing and model personalization are key strategies for addressing this issue. Clustered federated learning is a class of FL methods that groups clients that observe similarly distributed data into clusters, such that every client is typically associated with one data distribution and participates in training a model for that distribution along their cluster peers. In this paper, we present a unified Bayesian framework for clustered FL which associates clients to clusters. Then we propose several practical algorithms to handle the, otherwise growing, data associations in a way that trades off performance and computational complexity. This work provides insights on client-cluster associations and enables client knowledge sharing in new ways. The proposed framework circumvents the need for unique client-cluster associations, which is seen to increase the performance of the resulting models in a variety of experiments.

Paper Structure

This paper contains 30 sections, 19 equations, 13 figures, 11 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related works
  3. Personalized Federated Learning.
  4. Clustered Federated Learning.
  5. Bayesian clustered Federated Learning.
  6. CFL vs BCFL.
  7. Bayesian solution for clustered Federated Learning
  8. Bayesian clustered Federated Learning (BCFL)
  9. Problem formulation and conceptual solution.
  10. Decentralized computations.
  11. BCFL communication rounds as recursive updates
  12. The explosion of association hypotheses
  13. Approximate BCFL: simplified data association strategies
  14. Data association as an optimization problem
  15. Approximate BCFL algorithms
  16. ...and 15 more sections

Figures (13)

  • Figure 1: An example (top) association relation between clients and clusters; and details (bottom) of the operations and communications at the server and clients for BCFL.
  • Figure 2: Accuracies for AmazonReview (left panel) and Fashion-MNIST (right panel) datasets.
  • Figure 3: Client clustering during training for Digits-Five and selected CFL methods. $K=5$ clusters are pairwise split into $C=10$ clients, noted on the left labeling in (a).
  • Figure 4: Here we show a simple example of the assignment problem. The example of association with 2 clusters and 3 clients. The table is the loss of assignment from $C^j$ to $S^i$. The orange numbers are the optimal assignment loss. The optimal cost as the equation shows.
  • Figure 5: Warm-up comparison of accuracy for Digits-Five (left panel) and CIFAR-10 (right panel)
  • ...and 8 more figures