Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FedMoE: Personalized Federated Learning via Heterogeneous Mixture of Experts

Hanzi Mei, Dongqi Cai, Ao Zhou, Shangguang Wang, Mengwei Xu

TL;DR

FedMoE tackles heterogeneous Federated Learning for Large Language Models by integrating a sparsely activated Mixture-of-Experts backbone into a two-stage fine-tuning pipeline. The method first identifies client-specific sub-MoEs through activation-based heuristics and then performs federated training with modular aggregation and expert-recommendation-driven adjustments to refine structure and sharing. Empirical results show FedMoE improves task performance while reducing memory footprint and communication compared to baselines, and exhibits robust convergence across complex cross-task scenarios. The approach enables practical FedLLMs on edge devices by dynamically scaling capacity and sharing only the most relevant experts, thereby balancing personalization with global knowledge transfer.

Abstract

As Large Language Models (LLMs) push the boundaries of AI capabilities, their demand for data is growing. Much of this data is private and distributed across edge devices, making Federated Learning (FL) a de-facto alternative for fine-tuning (i.e., FedLLM). However, it faces significant challenges due to the inherent heterogeneity among clients, including varying data distributions and diverse task types. Towards a versatile FedLLM, we replace traditional dense model with a sparsely-activated Mixture-of-Experts (MoE) architecture, whose parallel feed-forward networks enable greater flexibility. To make it more practical in resource-constrained environments, we present FedMoE, the efficient personalized FL framework to address data heterogeneity, constructing an optimal sub-MoE for each client and bringing the knowledge back to global MoE. FedMoE is composed of two fine-tuning stages. In the first stage, FedMoE simplifies the problem by conducting a heuristic search based on observed activation patterns, which identifies a suboptimal submodel for each client. In the second stage, these submodels are distributed to clients for further training and returned for server aggregating through a novel modular aggregation strategy. Meanwhile, FedMoE progressively adjusts the submodels to optimal through global expert recommendation. Experimental results demonstrate the superiority of our method over previous personalized FL methods.

FedMoE: Personalized Federated Learning via Heterogeneous Mixture of Experts

TL;DR

FedMoE tackles heterogeneous Federated Learning for Large Language Models by integrating a sparsely activated Mixture-of-Experts backbone into a two-stage fine-tuning pipeline. The method first identifies client-specific sub-MoEs through activation-based heuristics and then performs federated training with modular aggregation and expert-recommendation-driven adjustments to refine structure and sharing. Empirical results show FedMoE improves task performance while reducing memory footprint and communication compared to baselines, and exhibits robust convergence across complex cross-task scenarios. The approach enables practical FedLLMs on edge devices by dynamically scaling capacity and sharing only the most relevant experts, thereby balancing personalization with global knowledge transfer.

Abstract

As Large Language Models (LLMs) push the boundaries of AI capabilities, their demand for data is growing. Much of this data is private and distributed across edge devices, making Federated Learning (FL) a de-facto alternative for fine-tuning (i.e., FedLLM). However, it faces significant challenges due to the inherent heterogeneity among clients, including varying data distributions and diverse task types. Towards a versatile FedLLM, we replace traditional dense model with a sparsely-activated Mixture-of-Experts (MoE) architecture, whose parallel feed-forward networks enable greater flexibility. To make it more practical in resource-constrained environments, we present FedMoE, the efficient personalized FL framework to address data heterogeneity, constructing an optimal sub-MoE for each client and bringing the knowledge back to global MoE. FedMoE is composed of two fine-tuning stages. In the first stage, FedMoE simplifies the problem by conducting a heuristic search based on observed activation patterns, which identifies a suboptimal submodel for each client. In the second stage, these submodels are distributed to clients for further training and returned for server aggregating through a novel modular aggregation strategy. Meanwhile, FedMoE progressively adjusts the submodels to optimal through global expert recommendation. Experimental results demonstrate the superiority of our method over previous personalized FL methods.
Paper Structure (28 sections, 6 equations, 4 figures, 2 tables, 1 algorithm)

This paper contains 28 sections, 6 equations, 4 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background and Motivation
  3. Federated Learning
  4. Mixture of Experts
  5. Motivation and Preliminary Experiments
  6. Method
  7. Problem Formulation
  8. Overview
  9. Model Structure
  10. Workflow
  11. Stage One: Coarse-grained Submodel Initialization
  12. Activation Probability Collection
  13. Heuristic Submodel Search
  14. Stage Two: Federated Training and Fine-grained Submodel Adjustment
  15. Submodel Deployment
  16. ...and 13 more sections

Figures (4)

  • Figure 1: Heatmap of expert activation frequencies across different training stages and datasets.
  • Figure 2: Performance of full model (8 experts per layer) and submodels (4 optimal experts per layer, 4 random experts per layer).
  • Figure 3: FedMoE workflow.
  • Figure 4: End-to-end performance of different personalized methods during training under Enforced-Hetero-T setting.