Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FedTrans: Efficient Federated Learning via Multi-Model Transformation

Yuxuan Zhu, Jiachen Liu, Mosharaf Chowdhury, Fan Lai

TL;DR

FedTrans tackles the challenge of heterogeneity in federated learning by automatically generating and training a suite of models of varying complexity for individual clients. It introduces a three-component framework—Model Transformer to morph bottleneck Cells, Client Manager to assign models via a utility-based, data-aware scheme, and Model Aggregator to softly share weights across models—to balance accuracy and training cost. Empirical results across four datasets show substantial per-client accuracy gains (up to 72%) and large cost reductions (up to 20x) compared to state-of-the-art multi-model FL methods, demonstrating strong scalability and practical impact for hardware-aware FL. The approach reduces manual model exploration and extends FL applicability to highly diverse client environments.

Abstract

Federated learning (FL) aims to train machine learning (ML) models across potentially millions of edge client devices. Yet, training and customizing models for FL clients is notoriously challenging due to the heterogeneity of client data, device capabilities, and the massive scale of clients, making individualized model exploration prohibitively expensive. State-of-the-art FL solutions personalize a globally trained model or concurrently train multiple models, but they often incur suboptimal model accuracy and huge training costs. In this paper, we introduce FedTrans, a multi-model FL training framework that automatically produces and trains high-accuracy, hardware-compatible models for individual clients at scale. FedTrans begins with a basic global model, identifies accuracy bottlenecks in model architectures during training, and then employs model transformation to derive new models for heterogeneous clients on the fly. It judiciously assigns models to individual clients while performing soft aggregation on multi-model updates to minimize total training costs. Our evaluations using realistic settings show that FedTrans improves individual client model accuracy by 14% - 72% while slashing training costs by 1.6X - 20X over state-of-the-art solutions.

FedTrans: Efficient Federated Learning via Multi-Model Transformation

TL;DR

FedTrans tackles the challenge of heterogeneity in federated learning by automatically generating and training a suite of models of varying complexity for individual clients. It introduces a three-component framework—Model Transformer to morph bottleneck Cells, Client Manager to assign models via a utility-based, data-aware scheme, and Model Aggregator to softly share weights across models—to balance accuracy and training cost. Empirical results across four datasets show substantial per-client accuracy gains (up to 72%) and large cost reductions (up to 20x) compared to state-of-the-art multi-model FL methods, demonstrating strong scalability and practical impact for hardware-aware FL. The approach reduces manual model exploration and extends FL applicability to highly diverse client environments.

Abstract

Federated learning (FL) aims to train machine learning (ML) models across potentially millions of edge client devices. Yet, training and customizing models for FL clients is notoriously challenging due to the heterogeneity of client data, device capabilities, and the massive scale of clients, making individualized model exploration prohibitively expensive. State-of-the-art FL solutions personalize a globally trained model or concurrently train multiple models, but they often incur suboptimal model accuracy and huge training costs. In this paper, we introduce FedTrans, a multi-model FL training framework that automatically produces and trains high-accuracy, hardware-compatible models for individual clients at scale. FedTrans begins with a basic global model, identifies accuracy bottlenecks in model architectures during training, and then employs model transformation to derive new models for heterogeneous clients on the fly. It judiciously assigns models to individual clients while performing soft aggregation on multi-model updates to minimize total training costs. Our evaluations using realistic settings show that FedTrans improves individual client model accuracy by 14% - 72% while slashing training costs by 1.6X - 20X over state-of-the-art solutions.
Paper Structure (48 sections, 4 equations, 15 figures, 7 tables, 1 algorithm)

This paper contains 48 sections, 4 equations, 15 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Motivation
  3. Challenges of Federated Learning
  4. System heterogeneity requires models with different complexities.
  5. Client accuracy variance implies no one-size-fits-all model.
  6. Limitations of Existing Solutions
  7. Huge training costs and laboring effort.
  8. Inferior model accuracy.
  9. FedTrans Overview
  10. System Components
  11. FedTrans Lifecycle
  12. FedTrans Design
  13. Model Transformer
  14. Identifying the right time to transform
  15. Picking the right model Cell to transform
  16. ...and 33 more sections

Figures (15)

  • Figure 1: Client system heterogeneity and accuracy variance call for different models for individual clients.
  • Figure 2: Existing solutions are suboptimal for FL clients.
  • Figure 3: FedTrans architecture and its lifecycle in FL deployment.
  • Figure 4: Model transformation on a convolution Cell.
  • Figure 5: Control flow of the transformation for a Cell$l$.
  • ...and 10 more figures