Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FedDebug: Systematic Debugging for Federated Learning Applications

Waris Gill, Ali Anwar, Muhammad Ali Gulzar

TL;DR

FedDebug addresses the challenge of debugging federated learning (FL) applications under strict data privacy by enabling interactive, replay-based debugging that does not access client data. It builds a simulation from selective telemetry, provides a breakpoint-based interface for stepwise inspection, and introduces an automatic fault-localization method using inference-guided test inputs and differential testing on neuron activations, followed by fix-and-replay to remove faulty contributions. The framework achieves 100% accuracy in locating a single faulty client and 90.3% for multiple faults across extensive FL configurations, with only about $1.2\%$ of a round’s training time in overhead. This approach offers a practical, privacy-preserving debugging pathway for real-world FL deployments and can be adapted to other FL frameworks beyond IBMFL.

Abstract

In Federated Learning (FL), clients independently train local models and share them with a central aggregator to build a global model. Impermissibility to access clients' data and collaborative training make FL appealing for applications with data-privacy concerns, such as medical imaging. However, these FL characteristics pose unprecedented challenges for debugging. When a global model's performance deteriorates, identifying the responsible rounds and clients is a major pain point. Developers resort to trial-and-error debugging with subsets of clients, hoping to increase the global model's accuracy or let future FL rounds retune the model, which are time-consuming and costly. We design a systematic fault localization framework, FedDebug, that advances the FL debugging on two novel fronts. First, FedDebug enables interactive debugging of realtime collaborative training in FL by leveraging record and replay techniques to construct a simulation that mirrors live FL. FedDebug's breakpoint can help inspect an FL state (round, client, and global model) and move between rounds and clients' models seamlessly, enabling a fine-grained step-by-step inspection. Second, FedDebug automatically identifies the client(s) responsible for lowering the global model's performance without any testing data and labels--both are essential for existing debugging techniques. FedDebug's strengths come from adapting differential testing in conjunction with neuron activations to determine the client(s) deviating from normal behavior. FedDebug achieves 100% accuracy in finding a single faulty client and 90.3% accuracy in finding multiple faulty clients. FedDebug's interactive debugging incurs 1.2% overhead during training, while it localizes a faulty client in only 2.1% of a round's training time.

FedDebug: Systematic Debugging for Federated Learning Applications

TL;DR

FedDebug addresses the challenge of debugging federated learning (FL) applications under strict data privacy by enabling interactive, replay-based debugging that does not access client data. It builds a simulation from selective telemetry, provides a breakpoint-based interface for stepwise inspection, and introduces an automatic fault-localization method using inference-guided test inputs and differential testing on neuron activations, followed by fix-and-replay to remove faulty contributions. The framework achieves 100% accuracy in locating a single faulty client and 90.3% for multiple faults across extensive FL configurations, with only about of a round’s training time in overhead. This approach offers a practical, privacy-preserving debugging pathway for real-world FL deployments and can be adapted to other FL frameworks beyond IBMFL.

Abstract

In Federated Learning (FL), clients independently train local models and share them with a central aggregator to build a global model. Impermissibility to access clients' data and collaborative training make FL appealing for applications with data-privacy concerns, such as medical imaging. However, these FL characteristics pose unprecedented challenges for debugging. When a global model's performance deteriorates, identifying the responsible rounds and clients is a major pain point. Developers resort to trial-and-error debugging with subsets of clients, hoping to increase the global model's accuracy or let future FL rounds retune the model, which are time-consuming and costly. We design a systematic fault localization framework, FedDebug, that advances the FL debugging on two novel fronts. First, FedDebug enables interactive debugging of realtime collaborative training in FL by leveraging record and replay techniques to construct a simulation that mirrors live FL. FedDebug's breakpoint can help inspect an FL state (round, client, and global model) and move between rounds and clients' models seamlessly, enabling a fine-grained step-by-step inspection. Second, FedDebug automatically identifies the client(s) responsible for lowering the global model's performance without any testing data and labels--both are essential for existing debugging techniques. FedDebug's strengths come from adapting differential testing in conjunction with neuron activations to determine the client(s) deviating from normal behavior. FedDebug achieves 100% accuracy in finding a single faulty client and 90.3% accuracy in finding multiple faulty clients. FedDebug's interactive debugging incurs 1.2% overhead during training, while it localizes a faulty client in only 2.1% of a round's training time.
Paper Structure (16 sections, 10 figures, 2 tables, 2 algorithms)

This paper contains 16 sections, 10 figures, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background and Motivation
  3. Federated Learning
  4. Motivating Scenario
  5. FedDebug's Debugging Constructs
  6. Selective Telemetry
  7. Interactive Replay Debugging
  8. Fix and Replay
  9. Faulty Client Localization
  10. Evaluation
  11. FedDebug's Performance
  12. Localization of Faulty Client(s)
  13. Neuron Activation Threshold
  14. Threats to Validity
  15. Related Work
  16. ...and 1 more sections

Figures (10)

  • Figure 1: In a centralized FL architecture, an aggregator sends a global model to clients (step 1). Each client trains the model on local data (step 2) and sends the locally trained model back to the server (step 3). The server aggregates all models to form a new global model (step 4).
  • Figure 2: Using FedDebug, a developer can set a breakpoint at round 20. When the FL application finishes round 20, FedDebug launches a Debugging Interface, reflected on the right. Step next (➋) takes the developer to the next step (round or client). Step-in increases the granularity of computation, e.g., round to client level. Resume (➌) rejoins the current execution status of the FL application if no intrusive actions are taken. At a given round, FedDebug can automatically localize the faulty client (➍) and then resume (➎) upon which the global model will be recomputed without the faulty client. This model will replace the corresponding round' s model, and FedDebug will start retraining from that round, round 22, in the FL interface.
  • Figure 3: An overview of FedDebug's fault localization approach. It first selects a random input that invokes diverse model behavior (A). It then applies differential execution on clients' models to localize a faulty client (B).
  • Figure 4: Global model (ResNet-34) prediction accuracy in the presence of a faulty client with different noise rates. Lower noise rates hardly degrade global model performance.
  • Figure 5: FedDebug's runtime overhead as a comparison between vanilla FL framework's aggregation time with FedDebug enabled FL aggregation.
  • ...and 5 more figures