Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Personalized Federated Learning via Feature Distribution Adaptation

Connor J. Mclaughlin, Lili Su

TL;DR

This work frames representation learning as a generative modeling task, where representations are trained with a classifier based on the global feature distribution, and proposes an algorithm, pFedFDA, that efficiently generates personalized models by adapting global generative classifiers to their local feature distributions.

Abstract

Federated learning (FL) is a distributed learning framework that leverages commonalities between distributed client datasets to train a global model. Under heterogeneous clients, however, FL can fail to produce stable training results. Personalized federated learning (PFL) seeks to address this by learning individual models tailored to each client. One approach is to decompose model training into shared representation learning and personalized classifier training. Nonetheless, previous works struggle to navigate the bias-variance trade-off in classifier learning, relying solely on limited local datasets or introducing costly techniques to improve generalization. In this work, we frame representation learning as a generative modeling task, where representations are trained with a classifier based on the global feature distribution. We then propose an algorithm, pFedFDA, that efficiently generates personalized models by adapting global generative classifiers to their local feature distributions. Through extensive computer vision benchmarks, we demonstrate that our method can adjust to complex distribution shifts with significant improvements over current state-of-the-art in data-scarce settings.

Personalized Federated Learning via Feature Distribution Adaptation

TL;DR

This work frames representation learning as a generative modeling task, where representations are trained with a classifier based on the global feature distribution, and proposes an algorithm, pFedFDA, that efficiently generates personalized models by adapting global generative classifiers to their local feature distributions.

Abstract

Federated learning (FL) is a distributed learning framework that leverages commonalities between distributed client datasets to train a global model. Under heterogeneous clients, however, FL can fail to produce stable training results. Personalized federated learning (PFL) seeks to address this by learning individual models tailored to each client. One approach is to decompose model training into shared representation learning and personalized classifier training. Nonetheless, previous works struggle to navigate the bias-variance trade-off in classifier learning, relying solely on limited local datasets or introducing costly techniques to improve generalization. In this work, we frame representation learning as a generative modeling task, where representations are trained with a classifier based on the global feature distribution. We then propose an algorithm, pFedFDA, that efficiently generates personalized models by adapting global generative classifiers to their local feature distributions. Through extensive computer vision benchmarks, we demonstrate that our method can adjust to complex distribution shifts with significant improvements over current state-of-the-art in data-scarce settings.

Paper Structure

This paper contains 27 sections, 1 theorem, 18 equations, 5 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Problem Formulation
  4. Methodology
  5. Generative Model of Feature Distributions
  6. Global Representation Learning
  7. Local Distribution Adaptation
  8. Experiments
  9. Experimental Setup
  10. Numerical Results
  11. Ablation of Method Components
  12. Communication and Computation
  13. Conclusion
  14. Limitations
  15. Broader Impacts
  16. ...and 12 more sections

Key Result

Theorem 1

Let $C=1$. Define $\mu_i$ as the sample mean of client $i$'s local features $\mu_i := \frac{1}{n_i}\sum_{j=1}^{n_i} z_{i}^j$, and $\mu_g$ as the global sample mean using all $N$ samples across $M$ clients: $\mu_g := \frac{1}{N}\sum_{i=1}^{M}\sum_{j=1}^{n_i} z_i^j$. Assume client features are indepen where $c>0$ is an absolute constant.

Figures (5)

  • Figure 1: Overview of pFedFDA. (Left) Heterogeneous clients collaboratively train representation parameters under a generative classifier derived from a global estimate of class feature distributions. (Right) At test time, clients adapt the generative classifier to their feature distributions to obtain personalized classifiers.
  • Figure 2: pFedFDA
  • Figure 3: Comparison of client $\beta$ and local dataset corruption on CIFAR10-S.
  • Figure 4: Comparison of Dirichlet Partitions on CIFAR10.
  • Figure 5: Comparison of average test accuracy with varying local epochs on CIFAR100.

Theorems & Definitions (2)

  • Theorem 1: Bias-Variance Trade-Off
  • proof : Proof of Theorem \ref{['Theorem:bias-variance']}