PatientDx: Merging Large Language Models for Protecting Data-Privacy in Healthcare
Jose G. Moreno, Jesus Lovon, M'Rick Robin-Charlet, Christine Damase-Michel, Lynda Tamine
TL;DR
PatientDx introduces a privacy-preserving approach to LLM deployment in healthcare by merging pre-trained models instead of fine-tuning on patient data. Using a Math-adapted pivotal model and merging strategies like Model Soup and SLerp, it achieves competitive mortality prediction on the MIMIC-IV dataset while reducing data leakage risks associated with fine-tuning. Empirical results indicate up to 7% AUROC improvement over input models and demonstrated transferability to unseen tasks, supported by qualitative QA and retrieval analyses. This work suggests that model merging can offer a practical, privacy-conscious path toward trustworthy, shareable healthcare LLMs, with future work focusing on more sophisticated merging strategies and broader task coverage.
Abstract
Fine-tuning of Large Language Models (LLMs) has become the default practice for improving model performance on a given task. However, performance improvement comes at the cost of training on vast amounts of annotated data which could be sensitive leading to significant data privacy concerns. In particular, the healthcare domain is one of the most sensitive domains exposed to data privacy issues. In this paper, we present PatientDx, a framework of model merging that allows the design of effective LLMs for health-predictive tasks without requiring fine-tuning nor adaptation on patient data. Our proposal is based on recently proposed techniques known as merging of LLMs and aims to optimize a building block merging strategy. PatientDx uses a pivotal model adapted to numerical reasoning and tunes hyperparameters on examples based on a performance metric but without training of the LLM on these data. Experiments using the mortality tasks of the MIMIC-IV dataset show improvements up to 7% in terms of AUROC when compared to initial models. Additionally, we confirm that when compared to fine-tuned models, our proposal is less prone to data leak problems without hurting performance. Finally, we qualitatively show the capabilities of our proposal through a case study. Our best model is publicly available at https://huggingface.co/ Jgmorenof/mistral\_merged\_0\_4.