Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Resource-Aware Heterogeneous Federated Learning using Neural Architecture Search

Sixing Yu, J. Pablo Muñoz, Ali Jannesari

TL;DR

Resource-aware Federated Learning (RaFL) tackles data non-IID and system heterogeneity by deploying resource-tailored neural architectures via a weight-sharing NAS supernet, enabled by on-device local knowledge fusion through deep mutual learning and cloud-level knowledge aggregation with optional ensemble distillation when public data is available. The approach combines on-demand NAS-derived subnetworks, a compact knowledge network for cross-client learning, and a cloud fusion strategy to integrate distributed knowledge, reducing communication overhead while enhancing inference efficiency at the edge. Empirical results on CIFAR-10/100 and FEMNIST across large-scale and sporadic FL scenarios show RaFL achieves superior learning and communication efficiency, high resource utilization, and robustness to heterogeneity compared to standard FL baselines and NAS/KD variants. These results indicate RaFL's practical potential for real-world deployments on diverse edge devices, and its framework supports transfer learning and scalable multi-architecture FL.

Abstract

Federated Learning (FL) is extensively used to train AI/ML models in distributed and privacy-preserving settings. Participant edge devices in FL systems typically contain non-independent and identically distributed (Non-IID) private data and unevenly distributed computational resources. Preserving user data privacy while optimizing AI/ML models in a heterogeneous federated network requires us to address data and system/resource heterogeneity. To address these challenges, we propose Resource-aware Federated Learning (RaFL). RaFL allocates resource-aware specialized models to edge devices using Neural Architecture Search (NAS) and allows heterogeneous model architecture deployment by knowledge extraction and fusion. Combining NAS and FL enables on-demand customized model deployment for resource-diverse edge devices. Furthermore, we propose a multi-model architecture fusion scheme allowing the aggregation of the distributed learning results. Results demonstrate RaFL's superior resource efficiency compared to SoTA.

Resource-Aware Heterogeneous Federated Learning using Neural Architecture Search

TL;DR

Resource-aware Federated Learning (RaFL) tackles data non-IID and system heterogeneity by deploying resource-tailored neural architectures via a weight-sharing NAS supernet, enabled by on-device local knowledge fusion through deep mutual learning and cloud-level knowledge aggregation with optional ensemble distillation when public data is available. The approach combines on-demand NAS-derived subnetworks, a compact knowledge network for cross-client learning, and a cloud fusion strategy to integrate distributed knowledge, reducing communication overhead while enhancing inference efficiency at the edge. Empirical results on CIFAR-10/100 and FEMNIST across large-scale and sporadic FL scenarios show RaFL achieves superior learning and communication efficiency, high resource utilization, and robustness to heterogeneity compared to standard FL baselines and NAS/KD variants. These results indicate RaFL's practical potential for real-world deployments on diverse edge devices, and its framework supports transfer learning and scalable multi-architecture FL.

Abstract

Federated Learning (FL) is extensively used to train AI/ML models in distributed and privacy-preserving settings. Participant edge devices in FL systems typically contain non-independent and identically distributed (Non-IID) private data and unevenly distributed computational resources. Preserving user data privacy while optimizing AI/ML models in a heterogeneous federated network requires us to address data and system/resource heterogeneity. To address these challenges, we propose Resource-aware Federated Learning (RaFL). RaFL allocates resource-aware specialized models to edge devices using Neural Architecture Search (NAS) and allows heterogeneous model architecture deployment by knowledge extraction and fusion. Combining NAS and FL enables on-demand customized model deployment for resource-diverse edge devices. Furthermore, we propose a multi-model architecture fusion scheme allowing the aggregation of the distributed learning results. Results demonstrate RaFL's superior resource efficiency compared to SoTA.
Paper Structure (16 sections, 9 equations, 7 figures, 4 tables)

This paper contains 16 sections, 9 equations, 7 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Methodology
  4. Resource-aware Federated Learning using Specialized Models
  5. Local Knowledge Fusion
  6. Cloud Knowledge Aggregation and Ensemble Distillation
  7. Experiments
  8. Experimental Setup
  9. Learning Efficiency
  10. Communication Efficiency
  11. Resource-aware System Heterogeneity
  12. Scaled and Sporadic FL
  13. Comparison with FL approaches that use Neural Architecture Search
  14. Comparison with KD-based FL
  15. Extra Burdens of Knowledge Networks
  16. ...and 1 more sections

Figures (7)

  • Figure 1: During initialization, edge clients request for specialized resource-aware models from the weight-sharing supernet with neural architecture search (NAS). During local training, the local resource-aware model and knowledge network are co-trained together via deep mutual learning. The server fuses the neural knowledge from clients and performs optional ensemble learning if public data is present.
  • Figure 2: Comparison of RaFL with SoTAs under vanilla aggregation setting.
  • Figure 3: Comparison of SoTAs with RaFL using (a) vanilla aggregation, (b) ensemble distillation and (c) transfer learning settings.
  • Figure 4: Performance comparison of RaFL and baselines.
  • Figure 5: Performance analysis of RaFL.
  • ...and 2 more figures