Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Revealing the influence of participant failures on model quality in cross-silo Federated Learning

Fabian Stricker, David Bermbach, Christian Zirpins

Abstract

Federated Learning (FL) is a paradigm for training machine learning (ML) models in collaborative settings while preserving participants' privacy by keeping raw data local. A key requirement for the use of FL in production is reliability, as insufficient reliability can compromise the validity, stability, and reproducibility of learning outcomes. FL inherently operates as a distributed system and is therefore susceptible to crash failures, network partitioning, and other fault scenarios. Despite this, the impact of such failures on FL outcomes has not yet been studied systematically. In this paper, we address this gap by investigating the impact of missing participants in FL. To this end, we conduct extensive experiments on image, tabular, and time-series data and analyze how the absence of participants affects model performance, taking into account influencing factors such as data skewness, different availability patterns, and model architectures. Furthermore, we examine scenario-specific aspects, including the utility of the global model for missing participants. Our experiments provide detailed insights into the effects of various influencing factors. In particular, we show that data skewness has a strong impact, often leading to overly optimistic model evaluations and, in some cases, even altering the effects of other influencing factors.

Revealing the influence of participant failures on model quality in cross-silo Federated Learning

Abstract

Federated Learning (FL) is a paradigm for training machine learning (ML) models in collaborative settings while preserving participants' privacy by keeping raw data local. A key requirement for the use of FL in production is reliability, as insufficient reliability can compromise the validity, stability, and reproducibility of learning outcomes. FL inherently operates as a distributed system and is therefore susceptible to crash failures, network partitioning, and other fault scenarios. Despite this, the impact of such failures on FL outcomes has not yet been studied systematically. In this paper, we address this gap by investigating the impact of missing participants in FL. To this end, we conduct extensive experiments on image, tabular, and time-series data and analyze how the absence of participants affects model performance, taking into account influencing factors such as data skewness, different availability patterns, and model architectures. Furthermore, we examine scenario-specific aspects, including the utility of the global model for missing participants. Our experiments provide detailed insights into the effects of various influencing factors. In particular, we show that data skewness has a strong impact, often leading to overly optimistic model evaluations and, in some cases, even altering the effects of other influencing factors.

Paper Structure

This paper contains 22 sections, 15 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background & Related Work
  3. Concepts of Federated Learning
  4. Related Work
  5. Participant Failures and Impact Modifiers
  6. Participant failures in FL systems
  7. Failure-impact modifiers
  8. Experiment Plan
  9. Experimentation Goals and Research Questions
  10. Experiment Design
  11. Datasets and Partitioning
  12. Model Architecture
  13. Contribution Measurement
  14. Simulating Participant Failures
  15. Performance Evaluation
  16. ...and 7 more sections

Figures (15)

  • Figure 1: Image Datasets: Comparison of the evaluation with and without the failed participant across base skew (BS) and manual skew (MS)
  • Figure 2: Tabular Datasets: Comparison of the evaluation with and without the failed participant for the BS and MS
  • Figure 3: GermanSolarFarm: Comparison of the evaluation with and without the missing participant across different models
  • Figure 4: CIFAR10 CNN: Different availability phases for the missing participant across different skews. The baseline represents the absence of the failure
  • Figure 5: Influence of the ResNet architecture on the availability phases for image datasets with BS. The baseline represents the absence of the failure
  • ...and 10 more figures