Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Generalized Meta Federated Learning Framework with Theoretical Convergence Guarantees

Mohammad Vahid Jamali, Hamid Saber, Jung Hyun Bae

TL;DR

This work extends meta-Federated Learning by optimizing the average device loss after an arbitrary number $\nu$ of local fine-tuning steps, introducing a generalized meta-FL objective $F(w)=\tfrac{1}{N}\sum_i f_i(\mathbf{w}^i_{\nu})$ and a FedAvg-like algorithm. It derives exact gradient expressions and proves convergence under standard assumptions, then develops practical first-order and Hessian-free approximations to enable scalable computation. Theoretical results characterize smoothness and convergence behavior with concrete bounds, including how the meta-loss propagates Hessian information across multiple updates. Empirical results on CIFAR-10/100 with heterogeneous client data show improved accuracy and faster convergence compared to Per-FedAvg and FedAvg, demonstrating the method’s potential for personalized FL in non-iid settings.

Abstract

Meta federated learning (FL) is a personalized variant of FL, where multiple agents collaborate on training an initial shared model without exchanging raw data samples. The initial model should be trained in a way that current or new agents can easily adapt it to their local datasets after one or a few fine-tuning steps, thus improving the model personalization. Conventional meta FL approaches minimize the average loss of agents on the local models obtained after one step of fine-tuning. In practice, agents may need to apply several fine-tuning steps to adapt the global model to their local data, especially under highly heterogeneous data distributions across agents. To this end, we present a generalized framework for the meta FL by minimizing the average loss of agents on their local model after any arbitrary number $ν$ of fine-tuning steps. For this generalized framework, we present a variant of the well-known federated averaging (FedAvg) algorithm and conduct a comprehensive theoretical convergence analysis to characterize the convergence speed as well as behavior of the meta loss functions in both the exact and approximated cases. Our experiments on real-world datasets demonstrate superior accuracy and faster convergence for the proposed scheme compared to conventional approaches.

A Generalized Meta Federated Learning Framework with Theoretical Convergence Guarantees

TL;DR

This work extends meta-Federated Learning by optimizing the average device loss after an arbitrary number of local fine-tuning steps, introducing a generalized meta-FL objective and a FedAvg-like algorithm. It derives exact gradient expressions and proves convergence under standard assumptions, then develops practical first-order and Hessian-free approximations to enable scalable computation. Theoretical results characterize smoothness and convergence behavior with concrete bounds, including how the meta-loss propagates Hessian information across multiple updates. Empirical results on CIFAR-10/100 with heterogeneous client data show improved accuracy and faster convergence compared to Per-FedAvg and FedAvg, demonstrating the method’s potential for personalized FL in non-iid settings.

Abstract

Meta federated learning (FL) is a personalized variant of FL, where multiple agents collaborate on training an initial shared model without exchanging raw data samples. The initial model should be trained in a way that current or new agents can easily adapt it to their local datasets after one or a few fine-tuning steps, thus improving the model personalization. Conventional meta FL approaches minimize the average loss of agents on the local models obtained after one step of fine-tuning. In practice, agents may need to apply several fine-tuning steps to adapt the global model to their local data, especially under highly heterogeneous data distributions across agents. To this end, we present a generalized framework for the meta FL by minimizing the average loss of agents on their local model after any arbitrary number of fine-tuning steps. For this generalized framework, we present a variant of the well-known federated averaging (FedAvg) algorithm and conduct a comprehensive theoretical convergence analysis to characterize the convergence speed as well as behavior of the meta loss functions in both the exact and approximated cases. Our experiments on real-world datasets demonstrate superior accuracy and faster convergence for the proposed scheme compared to conventional approaches.
Paper Structure (23 sections, 8 theorems, 146 equations, 4 figures)

This paper contains 23 sections, 8 theorems, 146 equations, 4 figures.

Table of Contents

  1. Introduction
  2. Generalized Meta FL under Multiple Fine-Tuning Steps
  3. General $\nu$ Fine-Tuning Steps
  4. Generalized Meta FL Algorithm
  5. Convergence Analysis
  6. Assumptions
  7. Several Useful Lemmas
  8. Convergence Result
  9. First-Order Approximations
  10. First-Order (FO) Approximation
  11. Hessian-Free (HF) Approximation
  12. Experiments
  13. Conclusion
  14. Closed Form of $\nabla F_i(\mathbf{w})$ for Arbitrary $\nu\in\{1,2,\cdots\}$
  15. Proof of Lemma \ref{['lemma_F_smooth']}
  16. ...and 8 more sections

Key Result

Lemma 4

Recall the definition of $F_i(\mathbf{w})$ in Meta_FLt2. If Assumptions asm_grad and asm_Hesian_Lip hold, then $F_i$ is smooth with parameter As a result, the average function $F(\mathbf{w})=\frac{1}{N}\sum_{i=1}^N F_i(\mathbf{w})$ is also smooth with parameter $L_F$.

Figures (4)

  • Figure 1: Accuracy $\%$ of various schemes over the CIFAR-10 dataset with $\alpha_d=0.01$.
  • Figure 2: Accuracy of various schemes over the CIFAR-100 dataset with $\alpha_d=0.01$.
  • Figure 3: Accuracy of various schemes over the CIFAR-10 dataset with $\alpha_d=0.01$.
  • Figure 4: Accuracy of our algorithms over the CIFAR-10 dataset with $\alpha_d=0.01$. Various batch sizes and numbers of fine-tuning steps are considered.

Theorems & Definitions (12)

  • Remark 1
  • Remark 2
  • Definition 3
  • Lemma 4
  • Lemma 5
  • Lemma 6
  • Lemma 7
  • Theorem 8
  • Remark 9
  • Lemma 10
  • ...and 2 more