HFedMoE: Resource-aware Heterogeneous Federated Learning with Mixture-of-Experts

TL;DR

HFedMoE tackles the challenges of resource-constrained, heterogeneous federated fine-tuning of Mixture-of-Experts language models by (1) identifying per-expert importance that informs client-specific expert selection, (2) employing an information-bottleneck framework to adaptively schedule a subset of experts within each device’s budget, and (3) applying sparsity-aware aggregation to align gating networks and selectively update active experts. The approach yields superior test accuracy and faster convergence under heterogeneous budgets, while maintaining robustness when computing resources are limited. It demonstrates practical impact by enabling efficient on-device LLM fine-tuning with MoE architectures, reducing computation and communication overhead without sacrificing global generalization. The findings suggest that coupling per-expert importance with budget-aware selection and focused aggregation can unlock scalable, personalized MoE-FL for large models in real-world edge environments.

Abstract

While federated learning (FL) enables fine-tuning of large language models (LLMs) without compromising data privacy, the substantial size of an LLM renders on-device training impractical for resource-constrained clients, such as mobile devices. Thus, Mixture-of-Experts (MoE) models have emerged as a computation-efficient solution, which activates only a sparse subset of experts during model training to reduce computing burden without sacrificing performance. Though integrating MoE into FL fine-tuning holds significant potential, it still encounters three key challenges: i) selecting appropriate experts for clients remains challenging due to the lack of a reliable metric to measure each expert's impact on local fine-tuning performance, ii) the heterogeneous computing resources across clients severely hinder MoE-based LLM fine-tuning, as dynamic expert activations across diverse input samples can overwhelm resource-constrained devices, and iii) client-specific expert subsets and routing preference undermine global aggregation, where misaligned expert updates and inconsistent gating networks in troduce destructive interference. To address these challenges, we propose HFedMoE, a heterogeneous MoE-based FL fine-tuning framework that customizes a subset of experts to each client for computation-efficient LLM fine-tuning. Specifically, HFedMoE identifies the expert importance based on its contributions to fine-tuning performance, and then adaptively selects a subset of experts from an information bottleneck perspective to align with each client' s computing budget. A sparsity-aware model aggregation strategy is also designed to aggregate the actively fine-tuned experts and gating parameters with importance weighted contributions. Extensive experiments demonstrate that HFedMoE outperforms state-of-the-art benchmarks in training accuracy and convergence speed.

Figures (16)

• Figure 1: The workflow for fine-tuning MoE-based LLM via FL across clients.
• Figure 2: The expert activations on AGNews dataset with global or local expert routing across clients, under a batch size of 4.
• Figure 3: The activated expert proportion within varying batch sizes and the converged performance (training round and test accuracy) under varying training failure rates.
• Figure 4: The activation frequency of experts and comparison of aggregation performance across clients on AGNews dataset.
• Figure 5: Overview of the HFedMoE framework. Each client quantifies per-expert importance within a training batch and selects a critical subset of experts for local fine-tuning under device-specific computing constraints. Only the activated experts and their gating networks are uploaded to the server, where a sparsity-aware aggregation strategy is applied to aggregate only active experts and explicitly align routing preferences across clients.