Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Federated Multi-Task Learning on Non-IID Data Silos: An Experimental Study

Yuwen Yang, Yuxiang Lu, Suizhi Huang, Shalayiding Sirejiding, Hongtao Lu, Yue Ding

TL;DR

This paper introduces FMTL-Bench, a comprehensive benchmark for Federated Multi-Task Learning (FMTL) on Non-IID data silos, integrating data, model, and optimization perspectives. It formalizes the FMTL objective, contrasts single-encoder multi-decoder (MD) and single-decoder conditioned (TC) architectures, and examines nine optimization baselines with parameter decoupling in federated contexts. Through seven data-partition experiments, multi-backbone model comparisons, and cross-domain scenarios, the authors analyze performance, scalability, and efficiency, including a case study on communication, time, and energy. Key insights show no universal winner across all settings, emphasize scenario-driven baseline selection, and highlight the benefits of pre-training and encoder-based decoupling for practical deployment, with code available for reproducibility.

Abstract

The innovative Federated Multi-Task Learning (FMTL) approach consolidates the benefits of Federated Learning (FL) and Multi-Task Learning (MTL), enabling collaborative model training on multi-task learning datasets. However, a comprehensive evaluation method, integrating the unique features of both FL and MTL, is currently absent in the field. This paper fills this void by introducing a novel framework, FMTL-Bench, for systematic evaluation of the FMTL paradigm. This benchmark covers various aspects at the data, model, and optimization algorithm levels, and comprises seven sets of comparative experiments, encapsulating a wide array of non-independent and identically distributed (Non-IID) data partitioning scenarios. We propose a systematic process for comparing baselines of diverse indicators and conduct a case study on communication expenditure, time, and energy consumption. Through our exhaustive experiments, we aim to provide valuable insights into the strengths and limitations of existing baseline methods, contributing to the ongoing discourse on optimal FMTL application in practical scenarios. The source code can be found on https://github.com/youngfish42/FMTL-Benchmark .

Federated Multi-Task Learning on Non-IID Data Silos: An Experimental Study

TL;DR

This paper introduces FMTL-Bench, a comprehensive benchmark for Federated Multi-Task Learning (FMTL) on Non-IID data silos, integrating data, model, and optimization perspectives. It formalizes the FMTL objective, contrasts single-encoder multi-decoder (MD) and single-decoder conditioned (TC) architectures, and examines nine optimization baselines with parameter decoupling in federated contexts. Through seven data-partition experiments, multi-backbone model comparisons, and cross-domain scenarios, the authors analyze performance, scalability, and efficiency, including a case study on communication, time, and energy. Key insights show no universal winner across all settings, emphasize scenario-driven baseline selection, and highlight the benefits of pre-training and encoder-based decoupling for practical deployment, with code available for reproducibility.

Abstract

The innovative Federated Multi-Task Learning (FMTL) approach consolidates the benefits of Federated Learning (FL) and Multi-Task Learning (MTL), enabling collaborative model training on multi-task learning datasets. However, a comprehensive evaluation method, integrating the unique features of both FL and MTL, is currently absent in the field. This paper fills this void by introducing a novel framework, FMTL-Bench, for systematic evaluation of the FMTL paradigm. This benchmark covers various aspects at the data, model, and optimization algorithm levels, and comprises seven sets of comparative experiments, encapsulating a wide array of non-independent and identically distributed (Non-IID) data partitioning scenarios. We propose a systematic process for comparing baselines of diverse indicators and conduct a case study on communication expenditure, time, and energy consumption. Through our exhaustive experiments, we aim to provide valuable insights into the strengths and limitations of existing baseline methods, contributing to the ongoing discourse on optimal FMTL application in practical scenarios. The source code can be found on https://github.com/youngfish42/FMTL-Benchmark .
Paper Structure (18 sections, 5 equations, 5 figures, 9 tables)

This paper contains 18 sections, 5 equations, 5 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Methodology Overview
  3. Experiment Evaluation
  4. Experimental Setup
  5. Comparative Experiments
  6. Data Level
  7. Model Level
  8. Optimization Algorithm Level
  9. Evaluation Criteria
  10. Case Study
  11. Additional Evaluation Criteria in the Case Study
  12. Results and Suggestions
  13. General Evaluation
  14. Data Level Analysis
  15. Model Level Analysis
  16. ...and 3 more sections

Figures (5)

  • Figure 1: Design of Comparative Experimental Scenarios in FMTL-Bench. Refer to (\ref{['sec: comp. exp']}) for detailed information.
  • Figure 2: Data Level: Relationship diagram and Data Distribution of Comparative Experiments in Federated Multi-Task Learning.
  • Figure 3: Case Study: Average task performance improvement (${\Delta\%\uparrow}$) versus communication rounds for baselines in IID-1 SDMT scenario from \ref{['tab: Comp Exp IID-1 and NIID-3']}. Please zoom in for details.
  • Figure 4: Case Study: Adjusted $p$-value heatmap for pairwise comparisons of baselines in IID-1 SDMT scenario from \ref{['tab: Comp Exp IID-1 and NIID-3']}.
  • Figure 5: Case Study: the Critical Difference (CD) Diagram for baselines in IID-1 SDMT scenario from Table \ref{['tab: Comp Exp IID-1 and NIID-3']}.