Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Personalized Federated Fine-Tuning of Vision Foundation Models for Healthcare

Adam Tupper, Christian Gagné

TL;DR

This work tackles privacy-preserving fine-tuning of vision foundation models for healthcare by proposing FedOPAL, a personalized federated fine-tuning approach using dual orthogonal LoRA adapters (global and personal). The global adapter learns general features shared across clients, while the personal adapter captures client-specific knowledge, with orthogonality regularization (either on weights or representations) to reduce interference. Empirical results on Fed-ISIC 2019 and Camelyon17-WILDS show competitive performance against baselines, highlighting the method's potential and the ongoing challenges of data heterogeneity and limited client sizes. If scaled further, FedOPAL could enable more effective, privacy-preserving collaboration in clinical settings through targeted, per-client model adaptation.

Abstract

Foundation models open up new possibilities for the use of AI in healthcare. However, even when pre-trained on health data, they still need to be fine-tuned for specific downstream tasks. Furthermore, although foundation models reduce the amount of training data required to achieve good performance, obtaining sufficient data is still a challenge. This is due, in part, to restrictions on sharing and aggregating data from different sources to protect patients' privacy. One possible solution to this is to fine-tune foundation models via federated learning across multiple participating clients (i.e., hospitals, clinics, etc.). In this work, we propose a new personalized federated fine-tuning method that learns orthogonal LoRA adapters to disentangle general and client-specific knowledge, enabling each client to fully exploit both their own data and the data of others. Our preliminary results on real-world federated medical imaging tasks demonstrate that our approach is competitive against current federated fine-tuning methods.

Personalized Federated Fine-Tuning of Vision Foundation Models for Healthcare

TL;DR

This work tackles privacy-preserving fine-tuning of vision foundation models for healthcare by proposing FedOPAL, a personalized federated fine-tuning approach using dual orthogonal LoRA adapters (global and personal). The global adapter learns general features shared across clients, while the personal adapter captures client-specific knowledge, with orthogonality regularization (either on weights or representations) to reduce interference. Empirical results on Fed-ISIC 2019 and Camelyon17-WILDS show competitive performance against baselines, highlighting the method's potential and the ongoing challenges of data heterogeneity and limited client sizes. If scaled further, FedOPAL could enable more effective, privacy-preserving collaboration in clinical settings through targeted, per-client model adaptation.

Abstract

Foundation models open up new possibilities for the use of AI in healthcare. However, even when pre-trained on health data, they still need to be fine-tuned for specific downstream tasks. Furthermore, although foundation models reduce the amount of training data required to achieve good performance, obtaining sufficient data is still a challenge. This is due, in part, to restrictions on sharing and aggregating data from different sources to protect patients' privacy. One possible solution to this is to fine-tune foundation models via federated learning across multiple participating clients (i.e., hospitals, clinics, etc.). In this work, we propose a new personalized federated fine-tuning method that learns orthogonal LoRA adapters to disentangle general and client-specific knowledge, enabling each client to fully exploit both their own data and the data of others. Our preliminary results on real-world federated medical imaging tasks demonstrate that our approach is competitive against current federated fine-tuning methods.
Paper Structure (8 sections, 2 equations, 3 figures, 2 tables)

This paper contains 8 sections, 2 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Personalized Orthogonal Adapters
  3. Experiments & Preliminary Results
  4. Conclusions and Future Work
  5. Datasets
  6. Fed-ISIC 2019
  7. Camelyon17-WILDS
  8. Experimental Setup

Figures (3)

  • Figure 1: An overview of FedOPAL. Left: Two trainable LoRA adapters (fire) are added to selected layers of a frozen pre-trained model (snowflake): a global adapter, collaboratively updated by all clients, and a personal adapter, which remains private to each client. Right: Orthogonality is enforced between either the LoRA $A$ matrix weights or the representations learned by each adapter to promote disentangled feature learning.
  • Figure 2: The client class distributions for the ISIC 2019 (left) and Camelyon17 (right) datasets.
  • Figure 3: Patches of normal tissue from each client (hospital) in the Camelyon17 dataset showing the colour variation due to the differences in staining.