Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Federated learning, ethics, and the double black box problem in medical AI

Joshua Hatherley, Anders Søgaard, Angela Ballantyne, Ruben Pauwels

TL;DR

This paper investigates ethical risks of medical federated learning (FL) and introduces federation opacity as a distinct epistemic challenge that creates a double black box in healthcare AI. It surveys FL fundamentals, healthcare applications, and expected benefits, then analyzes limitations in data security, data use, model performance, and algorithmic bias that may undercut claimed gains. The core contribution is articulating federation opacity and its implications for model security, performance, fairness, explainability, accountability, and the broader impact on care, scalability, and continual learning. The authors advocate for greater involvement of philosophers and humanities scholars to shape responsible FL design and governance in medicine.

Abstract

Federated learning (FL) is a machine learning approach that allows multiple devices or institutions to collaboratively train a model without sharing their local data with a third-party. FL is considered a promising way to address patient privacy concerns in medical artificial intelligence. The ethical risks of medical FL systems themselves, however, have thus far been underexamined. This paper aims to address this gap. We argue that medical FL presents a new variety of opacity -- federation opacity -- that, in turn, generates a distinctive double black box problem in healthcare AI. We highlight several instances in which the anticipated benefits of medical FL may be exaggerated, and conclude by highlighting key challenges that must be overcome to make FL ethically feasible in medicine.

Federated learning, ethics, and the double black box problem in medical AI

TL;DR

This paper investigates ethical risks of medical federated learning (FL) and introduces federation opacity as a distinct epistemic challenge that creates a double black box in healthcare AI. It surveys FL fundamentals, healthcare applications, and expected benefits, then analyzes limitations in data security, data use, model performance, and algorithmic bias that may undercut claimed gains. The core contribution is articulating federation opacity and its implications for model security, performance, fairness, explainability, accountability, and the broader impact on care, scalability, and continual learning. The authors advocate for greater involvement of philosophers and humanities scholars to shape responsible FL design and governance in medicine.

Abstract

Federated learning (FL) is a machine learning approach that allows multiple devices or institutions to collaboratively train a model without sharing their local data with a third-party. FL is considered a promising way to address patient privacy concerns in medical artificial intelligence. The ethical risks of medical FL systems themselves, however, have thus far been underexamined. This paper aims to address this gap. We argue that medical FL presents a new variety of opacity -- federation opacity -- that, in turn, generates a distinctive double black box problem in healthcare AI. We highlight several instances in which the anticipated benefits of medical FL may be exaggerated, and conclude by highlighting key challenges that must be overcome to make FL ethically feasible in medicine.
Paper Structure (22 sections, 1 figure)

This paper contains 22 sections, 1 figure.

Table of Contents

  1. Introduction
  2. What is federated learning?
  3. Federated learning in healthcare
  4. Applications
  5. Anticipated benefits
  6. Limitations
  7. Data security
  8. Data use
  9. Model performance
  10. Algorithmic bias
  11. The double black box problem
  12. Model security
  13. Model performance (again)
  14. Algorithmic bias (again)
  15. Explainability
  16. ...and 7 more sections

Figures (1)

  • Figure 1: The double black box problem. Adapted from yoo2022open.