Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Client Contribution Normalization for Enhanced Federated Learning

Mayank Kumar Kundalwal, Anurag Saraswat, Ishan Mishra, Deepak Mishra

TL;DR

This paper focuses on data-dependent heterogeneity in FL and proposes a novel approach leveraging mean latent representations extracted from locally trained models that enhances the global model's generalization and mitigates the limitations of conventional federated averaging methods.

Abstract

Mobile devices, including smartphones and laptops, generate decentralized and heterogeneous data, presenting significant challenges for traditional centralized machine learning models due to substantial communication costs and privacy risks. Federated Learning (FL) offers a promising alternative by enabling collaborative training of a global model across decentralized devices without data sharing. However, FL faces challenges due to statistical heterogeneity among clients, where non-independent and identically distributed (non-IID) data impedes model convergence and performance. This paper focuses on data-dependent heterogeneity in FL and proposes a novel approach leveraging mean latent representations extracted from locally trained models. The proposed method normalizes client contributions based on these representations, allowing the central server to estimate and adjust for heterogeneity during aggregation. This normalization enhances the global model's generalization and mitigates the limitations of conventional federated averaging methods. The main contributions include introducing a normalization scheme using mean latent representations to handle statistical heterogeneity in FL, demonstrating the seamless integration with existing FL algorithms to improve performance in non-IID settings, and validating the approach through extensive experiments on diverse datasets. Results show significant improvements in model accuracy and consistency across skewed distributions. Our experiments with six FL schemes: FedAvg, FedProx, FedBABU, FedNova, SCAFFOLD, and SGDM highlight the robustness of our approach. This research advances FL by providing a practical and computationally efficient solution for statistical heterogeneity, contributing to the development of more reliable and generalized machine learning models.

Client Contribution Normalization for Enhanced Federated Learning

TL;DR

This paper focuses on data-dependent heterogeneity in FL and proposes a novel approach leveraging mean latent representations extracted from locally trained models that enhances the global model's generalization and mitigates the limitations of conventional federated averaging methods.

Abstract

Mobile devices, including smartphones and laptops, generate decentralized and heterogeneous data, presenting significant challenges for traditional centralized machine learning models due to substantial communication costs and privacy risks. Federated Learning (FL) offers a promising alternative by enabling collaborative training of a global model across decentralized devices without data sharing. However, FL faces challenges due to statistical heterogeneity among clients, where non-independent and identically distributed (non-IID) data impedes model convergence and performance. This paper focuses on data-dependent heterogeneity in FL and proposes a novel approach leveraging mean latent representations extracted from locally trained models. The proposed method normalizes client contributions based on these representations, allowing the central server to estimate and adjust for heterogeneity during aggregation. This normalization enhances the global model's generalization and mitigates the limitations of conventional federated averaging methods. The main contributions include introducing a normalization scheme using mean latent representations to handle statistical heterogeneity in FL, demonstrating the seamless integration with existing FL algorithms to improve performance in non-IID settings, and validating the approach through extensive experiments on diverse datasets. Results show significant improvements in model accuracy and consistency across skewed distributions. Our experiments with six FL schemes: FedAvg, FedProx, FedBABU, FedNova, SCAFFOLD, and SGDM highlight the robustness of our approach. This research advances FL by providing a practical and computationally efficient solution for statistical heterogeneity, contributing to the development of more reliable and generalized machine learning models.

Paper Structure

This paper contains 17 sections, 8 equations, 2 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Statistical Heterogeneity in Federated Learning
  4. Methodology
  5. Mean Latent Representation
  6. Integration with Existing FL Algorithms
  7. Computational Overhead
  8. Normalization with Temperature Scaling
  9. Experimental Results
  10. Implementation Details
  11. Datasets and Data Distribution
  12. Results and Analysis
  13. CIFAR-10
  14. FEMNIST
  15. CINIC-10
  16. ...and 2 more sections

Figures (2)

  • Figure 1: Illustration of our proposed approach. Clients 1 and 2 have similar data distributions, while client 3 has a different data distribution. As a result, the locally trained model of client 3, denoted as $w^{c3}$, will be far from the models of the other two clients, $w^{c1}$ and $w^{c2}$, which are expected to be similar due to their similar data distributions. In the conventional FL setting, the same weight is assigned to all three clients during model aggregation, assuming that they have trained for the same number of epochs. However, in our proposed approach, the weight assigned to each client is influenced by the distance between that client's model and the models of other clients.
  • Figure 2: Convergence analysis of FL schemes on FEMNIST. Normalized approaches (solid lines) demonstrate consistently superior performance compared to original methods (dashed lines).