Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Survey on Model-heterogeneous Federated Learning: Problems, Methods, and Prospects

Boyu Fan, Siyang Jiang, Xiang Su, Sasu Tarkoma, Pan Hui

TL;DR

This paper reviews state-of-the-art approaches in model-heterogeneous FL, analyzing their strengths and weaknesses, while identifying open challenges and future research directions.

Abstract

As privacy concerns continue to grow, federated learning (FL) has gained significant attention as a promising privacy-preserving technology, leading to considerable advancements in recent years. Unlike traditional machine learning, which requires central data collection, FL keeps data localized on user devices. However, conventional FL assumes that all clients operate with identical model structures initialized by the server. In real-world applications, system heterogeneity is common, with clients possessing varying computational capabilities. This disparity can hinder training for resource-limited clients and result in inefficient resource use for those with greater processing power. To address this challenge, model-heterogeneous FL has been introduced, enabling clients to train models of varying complexity based on their hardware resources. This paper reviews state-of-the-art approaches in model-heterogeneous FL, analyzing their strengths and weaknesses, while identifying open challenges and future research directions. To the best of our knowledge, this is the first survey to specifically focus on model-heterogeneous FL.

A Survey on Model-heterogeneous Federated Learning: Problems, Methods, and Prospects

TL;DR

This paper reviews state-of-the-art approaches in model-heterogeneous FL, analyzing their strengths and weaknesses, while identifying open challenges and future research directions.

Abstract

As privacy concerns continue to grow, federated learning (FL) has gained significant attention as a promising privacy-preserving technology, leading to considerable advancements in recent years. Unlike traditional machine learning, which requires central data collection, FL keeps data localized on user devices. However, conventional FL assumes that all clients operate with identical model structures initialized by the server. In real-world applications, system heterogeneity is common, with clients possessing varying computational capabilities. This disparity can hinder training for resource-limited clients and result in inefficient resource use for those with greater processing power. To address this challenge, model-heterogeneous FL has been introduced, enabling clients to train models of varying complexity based on their hardware resources. This paper reviews state-of-the-art approaches in model-heterogeneous FL, analyzing their strengths and weaknesses, while identifying open challenges and future research directions. To the best of our knowledge, this is the first survey to specifically focus on model-heterogeneous FL.
Paper Structure (22 sections, 4 figures, 1 table)

This paper contains 22 sections, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Model-heterogeneous FL: Fundamental concepts and challenges
  3. Federated Learning
  4. System Heterogeneity
  5. Model-heterogeneous Federated Learning
  6. State-of-the-art Model-heterogeneous FL Approaches
  7. KD-based Approaches
  8. Knowledge Distillation
  9. Knowledge Distillation in FL
  10. Limitations
  11. PT-Based Approaches
  12. Partial Training
  13. Partial Training in FL
  14. Limitations
  15. Alternative Approaches
  16. ...and 7 more sections

Figures (4)

  • Figure 1: A standard FL architecture.
  • Figure 2: The overview of the challenge in model heterogeneity.
  • Figure 3: The illustration of KD in the model-heterogeneous FL.
  • Figure 4: The illustration of PT in the model-heterogeneous FL.