Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Flow of Knowledge: Federated Fine-Tuning of LLMs in Healthcare under Non-IID Conditions

Zeyu Chen, Yun Ji, Bowen Wang, Liwen Shi, Zijie Zeng, Sheng Zhang

TL;DR

The paper tackles privacy-preserving fine-tuning of large language models (LLMs) in healthcare under extreme non-IID data across institutions. It introduces a federated LoRA-driven framework with adapter-only updates, a central FedAvg-like aggregator, and a Cypherium-based blockchain identity and incentive layer, implemented on Ray for scalable multi-machine deployment. Empirical results show that Federated LoRA improves fairness (Min-Acc, H-mean) while maintaining overall accuracy across multilingual medical QA tasks and model scales, with a notable initial drop in the global model before convergence through multiple rounds. These findings demonstrate a practical pathway for privacy-preserving, cross-institutional healthcare AI collaboration and offer directions for extending the approach to other healthcare modalities and tasks.

Abstract

Large language models (LLMs) show great promise in healthcare, but their applications are hindered by data privacy restrictions and the challenges of cross-institution collaboration. Sensitive medical data cannot be centralized, while non-independent and identically distributed (non-IID) characteristics across institutions further complicate convergence and fairness. To address these issues, we present a federated fine-tuning approach based on Low-Rank Adaptation (LoRA), enabling privacy-preserving knowledge flow across institutions. The method iteratively combines local LoRA adaptation with global parameter aggregation, allowing efficient knowledge sharing without exposing raw data. A blockchain identity scheme is used for identifying individual LLM in such a distributed network. We evaluate this approach on heterogeneous and highly non-IID medical text datasets, where experiments demonstrate that federated LoRA not only enhances cross-client generalization but also improves the performance of the weakest client, achieving stable convergence and fairer outcomes. These findings highlight federated LoRA fine-tuning as a practical and effective paradigm for adapting LLMs in healthcare, offering a new path for multi-center medical AI collaboration.

Flow of Knowledge: Federated Fine-Tuning of LLMs in Healthcare under Non-IID Conditions

TL;DR

The paper tackles privacy-preserving fine-tuning of large language models (LLMs) in healthcare under extreme non-IID data across institutions. It introduces a federated LoRA-driven framework with adapter-only updates, a central FedAvg-like aggregator, and a Cypherium-based blockchain identity and incentive layer, implemented on Ray for scalable multi-machine deployment. Empirical results show that Federated LoRA improves fairness (Min-Acc, H-mean) while maintaining overall accuracy across multilingual medical QA tasks and model scales, with a notable initial drop in the global model before convergence through multiple rounds. These findings demonstrate a practical pathway for privacy-preserving, cross-institutional healthcare AI collaboration and offer directions for extending the approach to other healthcare modalities and tasks.

Abstract

Large language models (LLMs) show great promise in healthcare, but their applications are hindered by data privacy restrictions and the challenges of cross-institution collaboration. Sensitive medical data cannot be centralized, while non-independent and identically distributed (non-IID) characteristics across institutions further complicate convergence and fairness. To address these issues, we present a federated fine-tuning approach based on Low-Rank Adaptation (LoRA), enabling privacy-preserving knowledge flow across institutions. The method iteratively combines local LoRA adaptation with global parameter aggregation, allowing efficient knowledge sharing without exposing raw data. A blockchain identity scheme is used for identifying individual LLM in such a distributed network. We evaluate this approach on heterogeneous and highly non-IID medical text datasets, where experiments demonstrate that federated LoRA not only enhances cross-client generalization but also improves the performance of the weakest client, achieving stable convergence and fairer outcomes. These findings highlight federated LoRA fine-tuning as a practical and effective paradigm for adapting LLMs in healthcare, offering a new path for multi-center medical AI collaboration.

Paper Structure

This paper contains 17 sections, 4 equations, 2 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Proposed Methods
  3. Federated Fine-Tuning Workflow
  4. Base Models and Low-Rank Adaptation (LoRA)
  5. Base model selection
  6. LoRA principle and parameter efficiency
  7. Federated Aggregation
  8. Experiments and Results
  9. Federated Dataset Setup
  10. Experimental Setup
  11. Experimental Setup
  12. Base Experimental Setup
  13. Benchmarks and Evaluation
  14. Performance and Comparison
  15. Multi-round Aggregation Dynamics
  16. ...and 2 more sections

Figures (2)

  • Figure 1: Overview of our federated fine-tuning framework.
  • Figure 3: PCA visualization of LoRA parameter updates from three clients in the first communication round