Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mitigating Group Bias in Federated Learning for Heterogeneous Devices

Khotso Selialia, Yasra Chandio, Fatima M. Anwar

TL;DR

This work addresses group bias in Federated Learning caused by feature heterogeneity across heterogeneous edge devices. It introduces Multiplicative Weight Update with Regularization (MWR), which uses average conditional probabilities to derive cross-domain group importance weights from distributed data while preserving privacy. By combining a constrained optimization with $L1$ regularization and a normalization heuristic, MWR improves the worst-group performance without substantially degrading the best-group accuracy, achieving fairness across diverse client groups. Empirical evaluation on CIFAR10, DIGITS, MNIST, and Fashion-MNIST demonstrates that MWR substantially reduces group-bias and remains robust under varying feature noise and differential privacy budgets, underscoring its practical potential for fair, privacy-preserving FL in real-world, heterogeneous deployments.

Abstract

Federated Learning is emerging as a privacy-preserving model training approach in distributed edge applications. As such, most edge deployments are heterogeneous in nature i.e., their sensing capabilities and environments vary across deployments. This edge heterogeneity violates the independence and identical distribution (IID) property of local data across clients and produces biased global models i.e. models that contribute to unfair decision-making and discrimination against a particular community or a group. Existing bias mitigation techniques only focus on bias generated from label heterogeneity in non-IID data without accounting for domain variations due to feature heterogeneity and do not address global group-fairness property. Our work proposes a group-fair FL framework that minimizes group-bias while preserving privacy and without resource utilization overhead. Our main idea is to leverage average conditional probabilities to compute a cross-domain group \textit{importance weights} derived from heterogeneous training data to optimize the performance of the worst-performing group using a modified multiplicative weights update method. Additionally, we propose regularization techniques to minimize the difference between the worst and best-performing groups while making sure through our thresholding mechanism to strike a balance between bias reduction and group performance degradation. Our evaluation of human emotion recognition and image classification benchmarks assesses the fair decision-making of our framework in real-world heterogeneous settings.

Mitigating Group Bias in Federated Learning for Heterogeneous Devices

TL;DR

This work addresses group bias in Federated Learning caused by feature heterogeneity across heterogeneous edge devices. It introduces Multiplicative Weight Update with Regularization (MWR), which uses average conditional probabilities to derive cross-domain group importance weights from distributed data while preserving privacy. By combining a constrained optimization with regularization and a normalization heuristic, MWR improves the worst-group performance without substantially degrading the best-group accuracy, achieving fairness across diverse client groups. Empirical evaluation on CIFAR10, DIGITS, MNIST, and Fashion-MNIST demonstrates that MWR substantially reduces group-bias and remains robust under varying feature noise and differential privacy budgets, underscoring its practical potential for fair, privacy-preserving FL in real-world, heterogeneous deployments.

Abstract

Federated Learning is emerging as a privacy-preserving model training approach in distributed edge applications. As such, most edge deployments are heterogeneous in nature i.e., their sensing capabilities and environments vary across deployments. This edge heterogeneity violates the independence and identical distribution (IID) property of local data across clients and produces biased global models i.e. models that contribute to unfair decision-making and discrimination against a particular community or a group. Existing bias mitigation techniques only focus on bias generated from label heterogeneity in non-IID data without accounting for domain variations due to feature heterogeneity and do not address global group-fairness property. Our work proposes a group-fair FL framework that minimizes group-bias while preserving privacy and without resource utilization overhead. Our main idea is to leverage average conditional probabilities to compute a cross-domain group \textit{importance weights} derived from heterogeneous training data to optimize the performance of the worst-performing group using a modified multiplicative weights update method. Additionally, we propose regularization techniques to minimize the difference between the worst and best-performing groups while making sure through our thresholding mechanism to strike a balance between bias reduction and group performance degradation. Our evaluation of human emotion recognition and image classification benchmarks assesses the fair decision-making of our framework in real-world heterogeneous settings.
Paper Structure (22 sections, 6 equations, 8 figures, 4 tables, 1 algorithm)

This paper contains 22 sections, 6 equations, 8 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Bias in Machine Learning.
  4. Bias Metrics
  5. Bias Mitigation
  6. Preliminary Study
  7. Experimental Setup
  8. Key Findings
  9. Methodology
  10. Problem Statement and Workflow
  11. Enabling Privacy-preserving Group Fairness
  12. Ensuring Optimality through Regularization
  13. Bias Mitigation without Degrading High-Performing Groups
  14. Satisfying Performance Thresholds
  15. Evaluation
  16. ...and 7 more sections

Figures (8)

  • Figure 1: Illustrating the adverse effects of feature heterogeneity (noise) and its bias impact on image classification data lee2019context on an example language model (LM) in FL settings. The global LM, engaging in image captioning based on features from multiple clients, shows higher performance for images without distortions compared to those with a shift in feature distributions. This emphasizes the intricate interplay of feature heterogeneity and bias in FL, highlighting the influence of heterogeneous client datasets on the model's outcome.
  • Figure 2: Varied noise levels in CIFAR10 and DIGITS datasets. The notation "$\text{Noise}=x$" denotes the introduction of Gaussian noise with variance $x$," specifically applied to clients $D$ and $E$ in CIFAR10 and clients $A$ and $B$ in DIGITS.
  • Figure 3: Gradient distribution in a fully connected layer on the CIFAR10 dataset. The red and blue bars depict the local gradient distribution on client $1$ and client $2$, respectively. In (a), the distribution of local gradients is demonstrated across the two clients in IID settings. In (b), the distribution is shown in non-IID settings, with the introduction of Gaussian noise with variance $x$ ($\text{noise}=x$) on non-IID clients.
  • Figure 4: Overview of the proposed approach.
  • Figure 5: Examining the performance trade-off in $MWR$ concerning privacy and accuracy across various levels of differential privacy (DP) noise factors on FashionMNIST. In (a), a base Gaussian noise with a variance of $0.3$ is introduced to all methods, while in (b), Gaussian noise with a variance of $0.4$ is applied to all methods.
  • ...and 3 more figures