Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Hyper-parameter Optimization for Federated Learning with Step-wise Adaptive Mechanism

Yasaman Saadati, M. Hadi Amini

TL;DR

This paper investigates the deployment and integration of two lightweight Hyper-Parameter Optimization (HPO) tools, Raytune and Optuna, within the context of FL settings and introduces a novel client selection technique to mitigate the straggler effect in Auto-FL.

Abstract

Federated Learning (FL) is a decentralized learning approach that protects sensitive information by utilizing local model parameters rather than sharing clients' raw datasets. While this privacy-preserving method is widely employed across various applications, it still requires significant development and optimization. Automated Machine Learning (Auto-ML) has been adapted for reducing the need for manual adjustments. Previous studies have explored the integration of AutoML with different FL algorithms to evaluate their effectiveness in enhancing FL settings. However, Automated FL (Auto-FL) faces additional challenges due to the involvement of a large cohort of clients and global training rounds between clients and the server, rendering the tuning process time-consuming and nearly impossible on resource-constrained edge devices (e.g., IoT devices). This paper investigates the deployment and integration of two lightweight Hyper-Parameter Optimization (HPO) tools, Raytune and Optuna, within the context of FL settings. A step-wise feedback mechanism has also been designed to accelerate the hyper-parameter tuning process and coordinate AutoML toolkits with the FL server. To this end, both local and global feedback mechanisms are integrated to limit the search space and expedite the HPO process. Further, a novel client selection technique is introduced to mitigate the straggler effect in Auto-FL. The selected hyper-parameter tuning tools are evaluated using two benchmark datasets, FEMNIST, and CIFAR10. Further, the paper discusses the essential properties of successful HPO tools, the integration mechanism with the FL pipeline, and the challenges posed by the distributed and heterogeneous nature of FL environments.

Hyper-parameter Optimization for Federated Learning with Step-wise Adaptive Mechanism

TL;DR

This paper investigates the deployment and integration of two lightweight Hyper-Parameter Optimization (HPO) tools, Raytune and Optuna, within the context of FL settings and introduces a novel client selection technique to mitigate the straggler effect in Auto-FL.

Abstract

Federated Learning (FL) is a decentralized learning approach that protects sensitive information by utilizing local model parameters rather than sharing clients' raw datasets. While this privacy-preserving method is widely employed across various applications, it still requires significant development and optimization. Automated Machine Learning (Auto-ML) has been adapted for reducing the need for manual adjustments. Previous studies have explored the integration of AutoML with different FL algorithms to evaluate their effectiveness in enhancing FL settings. However, Automated FL (Auto-FL) faces additional challenges due to the involvement of a large cohort of clients and global training rounds between clients and the server, rendering the tuning process time-consuming and nearly impossible on resource-constrained edge devices (e.g., IoT devices). This paper investigates the deployment and integration of two lightweight Hyper-Parameter Optimization (HPO) tools, Raytune and Optuna, within the context of FL settings. A step-wise feedback mechanism has also been designed to accelerate the hyper-parameter tuning process and coordinate AutoML toolkits with the FL server. To this end, both local and global feedback mechanisms are integrated to limit the search space and expedite the HPO process. Further, a novel client selection technique is introduced to mitigate the straggler effect in Auto-FL. The selected hyper-parameter tuning tools are evaluated using two benchmark datasets, FEMNIST, and CIFAR10. Further, the paper discusses the essential properties of successful HPO tools, the integration mechanism with the FL pipeline, and the challenges posed by the distributed and heterogeneous nature of FL environments.

Paper Structure

This paper contains 15 sections, 1 equation, 4 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction:
  2. Related Work
  3. Hyper-parameter Optimization (HPO)
  4. Neural Architecture Search (NAS)
  5. Methodology
  6. Formulation of the Proposed HPO-FL Method
  7. Overview of the Proposed Framework and Deployed Tools
  8. Client selection in Auto-FL
  9. Experimental Setup
  10. Auto-ML tools
  11. Datasets
  12. Common Hyper-parameters of FL frameworks
  13. Implementation Details
  14. Experimental Result
  15. Conclusion and Future works

Figures (4)

  • Figure 1: Different stages of Auto-ML application.
  • Figure 2: Challenges of Automating the FL setting.
  • Figure 3: Overall Framework of Applying HPO in FL system. All local models used in this study and the majority of literature, are trained on the same DNN architecture but with different sets of HPs for evaluation. The HPO toolkit is inside the global trusted server.
  • Figure 4: The test accuracy of Random Search, Optuna, and RayTune HPO toolkits for both FEMNIST and CIFAR10 non-i.i.d datasets for 1000 communication rounds on (c),(d) small-scale(20 clients), and (a),(b) large-scale(200-clients) FL setting.