Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Federated Impression for Learning with Distributed Heterogeneous Data

Atrin Arya, Sana Ayromlou, Armin Saadat, Purang Abolmaesumi, Xiaoxiao Li

TL;DR

FedImpres is proposed which alleviates catastrophic forgetting by restoring synthetic data that represents the global information as federated impression to achieve this, and achieves state-of-the-art performance on both the BloodMNIST and Retina datasets.

Abstract

Standard deep learning-based classification approaches may not always be practical in real-world clinical applications, as they require a centralized collection of all samples. Federated learning (FL) provides a paradigm that can learn from distributed datasets across clients without requiring them to share data, which can help mitigate privacy and data ownership issues. In FL, sub-optimal convergence caused by data heterogeneity is common among data from different health centers due to the variety in data collection protocols and patient demographics across centers. Through experimentation in this study, we show that data heterogeneity leads to the phenomenon of catastrophic forgetting during local training. We propose FedImpres which alleviates catastrophic forgetting by restoring synthetic data that represents the global information as federated impression. To achieve this, we distill the global model resulting from each communication round. Subsequently, we use the synthetic data alongside the local data to enhance the generalization of local training. Extensive experiments show that the proposed method achieves state-of-the-art performance on both the BloodMNIST and Retina datasets, which contain label imbalance and domain shift, with an improvement in classification accuracy of up to 20%.

Federated Impression for Learning with Distributed Heterogeneous Data

TL;DR

FedImpres is proposed which alleviates catastrophic forgetting by restoring synthetic data that represents the global information as federated impression to achieve this, and achieves state-of-the-art performance on both the BloodMNIST and Retina datasets.

Abstract

Standard deep learning-based classification approaches may not always be practical in real-world clinical applications, as they require a centralized collection of all samples. Federated learning (FL) provides a paradigm that can learn from distributed datasets across clients without requiring them to share data, which can help mitigate privacy and data ownership issues. In FL, sub-optimal convergence caused by data heterogeneity is common among data from different health centers due to the variety in data collection protocols and patient demographics across centers. Through experimentation in this study, we show that data heterogeneity leads to the phenomenon of catastrophic forgetting during local training. We propose FedImpres which alleviates catastrophic forgetting by restoring synthetic data that represents the global information as federated impression. To achieve this, we distill the global model resulting from each communication round. Subsequently, we use the synthetic data alongside the local data to enhance the generalization of local training. Extensive experiments show that the proposed method achieves state-of-the-art performance on both the BloodMNIST and Retina datasets, which contain label imbalance and domain shift, with an improvement in classification accuracy of up to 20%.
Paper Structure (14 sections, 5 equations, 3 figures, 5 tables)

This paper contains 14 sections, 5 equations, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Problem setting
  4. Overview
  5. Federated Impression
  6. Forgetting-Penalized Local Training
  7. Experiments
  8. Datasets
  9. Experimental Settings
  10. Comparison with Baselines
  11. Ablation Studies
  12. Conclusion
  13. Acknowledgments.
  14. Disclosure of Interests.

Figures (3)

  • Figure 1: Catastrophic forgetting occurs when server weights are overwritten during local training, causing a loss of previous knowledge. To investigate the effect of catastrophic forgetting during local training in FL, we conducted experiments on BloodMNIST using the same experimental settings described in Sec. \ref{['sec:exp']}. Specifically, we plot each client's local model accuracy over other clients' data during local training. The accuracy drops drastically using FedAvg; however, FedImpres maintains stable accuracy across clients.
  • Figure 2: Our proposed approach, FedImpres, aims to capture the global distribution learned by the aggregated server model and distill it into a dataset that can be shared with clients. The approach consists of two steps: a) First, we perform pixel-wise optimization by starting from unlabeled public data and using the server model's predicted pseudo-labels to backpropagate using Eq.(\ref{['FI']}). b) Second, to improve local training, we add the synthesized data as a regularizer to the local data using Eq.(\ref{['CE']}). This allows us to share the global distilled distribution with clients and leverage it to improve local training.
  • Figure 3: Top: BloodMNIST, consisting of eight classes; bottom: Normal (class 1) and Glaucomatous (class 2) images from the Retina, collected from each site.