Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving Federated Learning Personalization via Model Agnostic Meta Learning

Yihan Jiang, Jakub Konečný, Keith Rush, Sreeram Kannan

TL;DR

The paper reframes federated learning as a natural arena for meta-learning, showing FedAvg implicitly performs meta-learning updates and that personalization can be enhanced with a two-stage approach. It introduces Personalized FedAvg, combining FedAvg with a fine-tuning stage (Reptile(K) and Adam) to optimize both personalized performance and the quality of the initial model. Empirical results on EMNIST-62 and Shakespeare demonstrate improved personalized accuracy and stability, while also revealing that increasing local updates can boost personalization up to a limit. The work challenges the conventional FL objective of global accuracy alone and connects FL with MAML, suggesting new directions for optimization and evaluation in both fields.

Abstract

Federated Learning (FL) refers to learning a high quality global model based on decentralized data storage, without ever copying the raw data. A natural scenario arises with data created on mobile phones by the activity of their users. Given the typical data heterogeneity in such situations, it is natural to ask how can the global model be personalized for every such device, individually. In this work, we point out that the setting of Model Agnostic Meta Learning (MAML), where one optimizes for a fast, gradient-based, few-shot adaptation to a heterogeneous distribution of tasks, has a number of similarities with the objective of personalization for FL. We present FL as a natural source of practical applications for MAML algorithms, and make the following observations. 1) The popular FL algorithm, Federated Averaging, can be interpreted as a meta learning algorithm. 2) Careful fine-tuning can yield a global model with higher accuracy, which is at the same time easier to personalize. However, solely optimizing for the global model accuracy yields a weaker personalization result. 3) A model trained using a standard datacenter optimization method is much harder to personalize, compared to one trained using Federated Averaging, supporting the first claim. These results raise new questions for FL, MAML, and broader ML research.

Improving Federated Learning Personalization via Model Agnostic Meta Learning

TL;DR

The paper reframes federated learning as a natural arena for meta-learning, showing FedAvg implicitly performs meta-learning updates and that personalization can be enhanced with a two-stage approach. It introduces Personalized FedAvg, combining FedAvg with a fine-tuning stage (Reptile(K) and Adam) to optimize both personalized performance and the quality of the initial model. Empirical results on EMNIST-62 and Shakespeare demonstrate improved personalized accuracy and stability, while also revealing that increasing local updates can boost personalization up to a limit. The work challenges the conventional FL objective of global accuracy alone and connects FL with MAML, suggesting new directions for optimization and evaluation in both fields.

Abstract

Federated Learning (FL) refers to learning a high quality global model based on decentralized data storage, without ever copying the raw data. A natural scenario arises with data created on mobile phones by the activity of their users. Given the typical data heterogeneity in such situations, it is natural to ask how can the global model be personalized for every such device, individually. In this work, we point out that the setting of Model Agnostic Meta Learning (MAML), where one optimizes for a fast, gradient-based, few-shot adaptation to a heterogeneous distribution of tasks, has a number of similarities with the objective of personalization for FL. We present FL as a natural source of practical applications for MAML algorithms, and make the following observations. 1) The popular FL algorithm, Federated Averaging, can be interpreted as a meta learning algorithm. 2) Careful fine-tuning can yield a global model with higher accuracy, which is at the same time easier to personalize. However, solely optimizing for the global model accuracy yields a weaker personalization result. 3) A model trained using a standard datacenter optimization method is much harder to personalize, compared to one trained using Federated Averaging, supporting the first claim. These results raise new questions for FL, MAML, and broader ML research.

Paper Structure

This paper contains 12 sections, 5 equations, 4 figures, 3 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Interpreting FedAvg as a Meta Learning Algorithm
  3. Personalized FedAvg
  4. Convergence of FedAvg
  5. Behavior of Personalization
  6. Discussion and Future Work
  7. Challenges for Federated Learning.
  8. Challenges for Model Agnostic Meta Learning.
  9. Challenges for broader Machine Learning.
  10. Appendix
  11. Fine Tuning Optimizers
  12. Per-Client Personalization Results

Figures (4)

  • Figure 1: Training Convergence on EMNIST-62 (left) and Shakespeare (right).
  • Figure 2: Personalized accuracy as a function of local update epochs. Federated initial models (left), initial models trained by centralizing data and using default Adam optimizer (right).
  • Figure 3: Distribution of initial and personalized accuracies of fine tuned models.
  • Figure 4: Performance for sample of test clients for EMNIST-62 (left) and Shakespeare (right)