Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Federated Fairness Analytics: Quantifying Fairness in Federated Learning

Oscar Dilley, Juan Marcelo Parra-Ullauri, Rasheed Hussain, Dimitra Simeonidou

TL;DR

Using the techniques of Federated Fairness Analytics, FL practitioners can uncover previously unobtainable insights into their system's fairness, at differing levels of granularity in order to address fairness challenges in FL.

Abstract

Federated Learning (FL) is a privacy-enhancing technology for distributed ML. By training models locally and aggregating updates - a federation learns together, while bypassing centralised data collection. FL is increasingly popular in healthcare, finance and personal computing. However, it inherits fairness challenges from classical ML and introduces new ones, resulting from differences in data quality, client participation, communication constraints, aggregation methods and underlying hardware. Fairness remains an unresolved issue in FL and the community has identified an absence of succinct definitions and metrics to quantify fairness; to address this, we propose Federated Fairness Analytics - a methodology for measuring fairness. Our definition of fairness comprises four notions with novel, corresponding metrics. They are symptomatically defined and leverage techniques originating from XAI, cooperative game-theory and networking engineering. We tested a range of experimental settings, varying the FL approach, ML task and data settings. The results show that statistical heterogeneity and client participation affect fairness and fairness conscious approaches such as Ditto and q-FedAvg marginally improve fairness-performance trade-offs. Using our techniques, FL practitioners can uncover previously unobtainable insights into their system's fairness, at differing levels of granularity in order to address fairness challenges in FL. We have open-sourced our work at: https://github.com/oscardilley/federated-fairness.

Federated Fairness Analytics: Quantifying Fairness in Federated Learning

TL;DR

Using the techniques of Federated Fairness Analytics, FL practitioners can uncover previously unobtainable insights into their system's fairness, at differing levels of granularity in order to address fairness challenges in FL.

Abstract

Federated Learning (FL) is a privacy-enhancing technology for distributed ML. By training models locally and aggregating updates - a federation learns together, while bypassing centralised data collection. FL is increasingly popular in healthcare, finance and personal computing. However, it inherits fairness challenges from classical ML and introduces new ones, resulting from differences in data quality, client participation, communication constraints, aggregation methods and underlying hardware. Fairness remains an unresolved issue in FL and the community has identified an absence of succinct definitions and metrics to quantify fairness; to address this, we propose Federated Fairness Analytics - a methodology for measuring fairness. Our definition of fairness comprises four notions with novel, corresponding metrics. They are symptomatically defined and leverage techniques originating from XAI, cooperative game-theory and networking engineering. We tested a range of experimental settings, varying the FL approach, ML task and data settings. The results show that statistical heterogeneity and client participation affect fairness and fairness conscious approaches such as Ditto and q-FedAvg marginally improve fairness-performance trade-offs. Using our techniques, FL practitioners can uncover previously unobtainable insights into their system's fairness, at differing levels of granularity in order to address fairness challenges in FL. We have open-sourced our work at: https://github.com/oscardilley/federated-fairness.
Paper Structure (25 sections, 8 equations, 8 figures, 3 tables)

This paper contains 25 sections, 8 equations, 8 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Technical Background
  3. Related Work
  4. Proposed Techniques to Quantify Fairness
  5. Experimental Design
  6. Results and Discussion
  7. Conclusion
  8. Supplementary Material
  9. Acknowledgements
  10. Author contributions statement
  11. Data Availability Statement
  12. Additional information

Figures (8)

  • Figure 1: The client-server architecture of a typical FL system with a central server and clients that exhibit statistical heterogeneity due to differing datasets.
  • Figure 2: A logical schematic of the simulation testbed.
  • Figure 3: Sample results at the fairness-metric level of abstraction.
  • Figure 4: Comparing the general fairness of different approaches against the mean-average performance.
  • Figure 5: CIFAR-10 Cross-Device Simulation Results.
  • ...and 3 more figures