Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Training Heterogeneous Client Models using Knowledge Distillation in Serverless Federated Learning

Mohak Chadha, Pulkit Khera, Jianfeng Gu, Osama Abboud, Michael Gerndt

TL;DR

This work tackles the challenge of training heterogeneous client models in serverless federated learning by leveraging knowledge distillation (KD). It introduces optimized serverless workflows for two KD strategies, FedMD and FedDF, implemented within the FedLess framework to support diverse client architectures and data distributions. Through a comprehensive distributed evaluation across multiple datasets and non-IID settings, the authors demonstrate that serverless FedDF provides superior robustness, speed, and lower costs compared to FedMD, while their system optimizations yield substantial speedups (up to 3.5x for pretraining and 1.76x for distillation). The results have practical implications for scalable, cost-effective FL deployments on serverless platforms, and point to future directions in data-free KD and more flexible, platform-agnostic KD pipelines.

Abstract

Federated Learning (FL) is an emerging machine learning paradigm that enables the collaborative training of a shared global model across distributed clients while keeping the data decentralized. Recent works on designing systems for efficient FL have shown that utilizing serverless computing technologies, particularly Function-as-a-Service (FaaS) for FL, can enhance resource efficiency, reduce training costs, and alleviate the complex infrastructure management burden on data holders. However, existing serverless FL systems implicitly assume a uniform global model architecture across all participating clients during training. This assumption fails to address fundamental challenges in practical FL due to the resource and statistical data heterogeneity among FL clients. To address these challenges and enable heterogeneous client models in serverless FL, we utilize Knowledge Distillation (KD) in this paper. Towards this, we propose novel optimized serverless workflows for two popular conventional federated KD techniques, i.e., FedMD and FedDF. We implement these workflows by introducing several extensions to an open-source serverless FL system called FedLess. Moreover, we comprehensively evaluate the two strategies on multiple datasets across varying levels of client data heterogeneity using heterogeneous client models with respect to accuracy, fine-grained training times, and costs. Results from our experiments demonstrate that serverless FedDF is more robust to extreme non-IID data distributions, is faster, and leads to lower costs than serverless FedMD. In addition, compared to the original implementation, our optimizations for particular steps in FedMD and FedDF lead to an average speedup of 3.5x and 1.76x across all datasets.

Training Heterogeneous Client Models using Knowledge Distillation in Serverless Federated Learning

TL;DR

This work tackles the challenge of training heterogeneous client models in serverless federated learning by leveraging knowledge distillation (KD). It introduces optimized serverless workflows for two KD strategies, FedMD and FedDF, implemented within the FedLess framework to support diverse client architectures and data distributions. Through a comprehensive distributed evaluation across multiple datasets and non-IID settings, the authors demonstrate that serverless FedDF provides superior robustness, speed, and lower costs compared to FedMD, while their system optimizations yield substantial speedups (up to 3.5x for pretraining and 1.76x for distillation). The results have practical implications for scalable, cost-effective FL deployments on serverless platforms, and point to future directions in data-free KD and more flexible, platform-agnostic KD pipelines.

Abstract

Federated Learning (FL) is an emerging machine learning paradigm that enables the collaborative training of a shared global model across distributed clients while keeping the data decentralized. Recent works on designing systems for efficient FL have shown that utilizing serverless computing technologies, particularly Function-as-a-Service (FaaS) for FL, can enhance resource efficiency, reduce training costs, and alleviate the complex infrastructure management burden on data holders. However, existing serverless FL systems implicitly assume a uniform global model architecture across all participating clients during training. This assumption fails to address fundamental challenges in practical FL due to the resource and statistical data heterogeneity among FL clients. To address these challenges and enable heterogeneous client models in serverless FL, we utilize Knowledge Distillation (KD) in this paper. Towards this, we propose novel optimized serverless workflows for two popular conventional federated KD techniques, i.e., FedMD and FedDF. We implement these workflows by introducing several extensions to an open-source serverless FL system called FedLess. Moreover, we comprehensively evaluate the two strategies on multiple datasets across varying levels of client data heterogeneity using heterogeneous client models with respect to accuracy, fine-grained training times, and costs. Results from our experiments demonstrate that serverless FedDF is more robust to extreme non-IID data distributions, is faster, and leads to lower costs than serverless FedMD. In addition, compared to the original implementation, our optimizations for particular steps in FedMD and FedDF lead to an average speedup of 3.5x and 1.76x across all datasets.
Paper Structure (28 sections, 10 figures, 8 tables)

This paper contains 28 sections, 10 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Knowledge Distillation in FL
  4. FedMD
  5. FedDF
  6. Related Work
  7. Serverless Federated Learning
  8. Model-agnostic Serverless FL
  9. System Design
  10. Supporting Heterogeneous Client Model Architectures
  11. Extending FedLess
  12. Serverless Parallel Transfer Learning for FedMD
  13. Serverless Parallel Ensemble Distillation for FedDF
  14. Serverless Knowledge Distillation
  15. Serverless FedMD
  16. ...and 13 more sections

Figures (10)

  • Figure 1: Global model accuracy across different model architectures on the CIFAR-10 dataset with FedAvgmcmahan2017communication for both IID and non-IID data distributions. The models are trained with FedLessfedless for $100$ clients.
  • Figure 2: Synchronous conventional KD and serverless KD.
  • Figure 3: Different steps in the FedMDfedmd algorithm.
  • Figure 4: Different steps in the FedDFfeddf algorithm.
  • Figure 5: Proposed extensions (green) to FedLessfedless.
  • ...and 5 more figures