Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FEDKIM: Adaptive Federated Knowledge Injection into Medical Foundation Models

Xiaochen Wang, Jiaqi Wang, Houping Xiao, Jinghui Chen, Fenglong Ma

TL;DR

Medical foundation models are limited by data privacy constraints and modality coverage. FedKIM delivers privacy-preserving knowledge injection by federating knowledge extraction from private client data to a server-side foundation model, and injecting it through an adaptive Multimodal Multitask Mixture Of Experts (M3OE) with LoRA-based parameter-efficient fine-tuning. The framework combines client-side encoders and a server-side injector, enabling standard FL aggregation and public-data alignment, and is evaluated on 12 tasks across 7 modalities with both zero-shot and fine-tuning scenarios, plus ablations. Results show FedKIM improves cross-modality generalization and unseen-task performance, demonstrating a practical path to scalable, privacy-preserving medical foundation models.

Abstract

Foundation models have demonstrated remarkable capabilities in handling diverse modalities and tasks, outperforming conventional artificial intelligence (AI) approaches that are highly task-specific and modality-reliant. In the medical domain, however, the development of comprehensive foundation models is constrained by limited access to diverse modalities and stringent privacy regulations. To address these constraints, this study introduces a novel knowledge injection approach, FedKIM, designed to scale the medical foundation model within a federated learning framework. FedKIM leverages lightweight local models to extract healthcare knowledge from private data and integrates this knowledge into a centralized foundation model using a designed adaptive Multitask Multimodal Mixture Of Experts (M3OE) module. This method not only preserves privacy but also enhances the model's ability to handle complex medical tasks involving multiple modalities. Our extensive experiments across twelve tasks in seven modalities demonstrate the effectiveness of FedKIM in various settings, highlighting its potential to scale medical foundation models without direct access to sensitive data.

FEDKIM: Adaptive Federated Knowledge Injection into Medical Foundation Models

TL;DR

Medical foundation models are limited by data privacy constraints and modality coverage. FedKIM delivers privacy-preserving knowledge injection by federating knowledge extraction from private client data to a server-side foundation model, and injecting it through an adaptive Multimodal Multitask Mixture Of Experts (M3OE) with LoRA-based parameter-efficient fine-tuning. The framework combines client-side encoders and a server-side injector, enabling standard FL aggregation and public-data alignment, and is evaluated on 12 tasks across 7 modalities with both zero-shot and fine-tuning scenarios, plus ablations. Results show FedKIM improves cross-modality generalization and unseen-task performance, demonstrating a practical path to scalable, privacy-preserving medical foundation models.

Abstract

Foundation models have demonstrated remarkable capabilities in handling diverse modalities and tasks, outperforming conventional artificial intelligence (AI) approaches that are highly task-specific and modality-reliant. In the medical domain, however, the development of comprehensive foundation models is constrained by limited access to diverse modalities and stringent privacy regulations. To address these constraints, this study introduces a novel knowledge injection approach, FedKIM, designed to scale the medical foundation model within a federated learning framework. FedKIM leverages lightweight local models to extract healthcare knowledge from private data and integrates this knowledge into a centralized foundation model using a designed adaptive Multitask Multimodal Mixture Of Experts (M3OE) module. This method not only preserves privacy but also enhances the model's ability to handle complex medical tasks involving multiple modalities. Our extensive experiments across twelve tasks in seven modalities demonstrate the effectiveness of FedKIM in various settings, highlighting its potential to scale medical foundation models without direct access to sensitive data.
Paper Structure (27 sections, 6 equations, 2 figures, 8 tables)

This paper contains 27 sections, 6 equations, 2 figures, 8 tables.

Table of Contents

  1. Introduction
  2. The proposed FedKIM Framework
  3. Framework Setups
  4. Client Setups
  5. Server Setups
  6. Client Updates -- Knowledge Extraction from Private Clients
  7. Server Updates -- Knowledge Injection into Medical LLM
  8. Knowledge Aggregation
  9. Knowledge Injection
  10. Experiment Setup
  11. Task Introduction
  12. Data Partition
  13. Baselines
  14. FL Backbone Approaches
  15. Implementation Details
  16. ...and 12 more sections

Figures (2)

  • Figure 1: Illustration of the proposed FedKIM. (a) Framework overview, where the proposed FedKIM contains client and server updates. (b) Federated knowledge injection, where FedKIM first aggregates models uploaded from clients and then injects the aggregated model knowledge into medical foundation model $\mathcal{F}$ with three steps. "PEFT" in Step 3 denotes parameter-efficient fine-tuning.
  • Figure 2: Performance comparison between FedKIM and baselines on the zero-shot evaluation.