Generalization of Medical Large Language Models through Cross-Domain Weak Supervision
Robert Long, Eric Gonzalez, Harrison Fuller
TL;DR
The paper tackles the challenge of adapting general-purpose LLMs to the medical domain while mitigating catastrophic forgetting. It proposes Incremental Curriculum-Based Fine-Tuning (ICFT), a framework that merges curriculum learning, dual-stage memory coordination between Short-Term Memory (STM) and Long-Term Memory (LTM), and parameter-efficient fine-tuning via Low-Rank Adaptation (LoRA). The training objective combines $L_{ ext{consistency}}$, $L_{ ext{task}}$, and $L_{ ext{fine-tune}}$ into $L_{ ext{total}} = L_{ ext{consistency}} + L_{ ext{task}} + L_{ ext{fine-tune}}$, and experiments on QA, preference classification, and response generation show superior accuracy, efficiency, and generalization. Results indicate strong potential for real-world healthcare deployment and suggest future extension to multi-modal data.
Abstract
The advancement of large language models (LLMs) has opened new frontiers in natural language processing, particularly in specialized domains like healthcare. In this paper, we propose the Incremental Curriculum-Based Fine-Tuning (ICFT) framework to enhance the generative capabilities of medical large language models (MLLMs). ICFT combines curriculum-based learning, dual-stage memory coordination, and parameter-efficient fine-tuning to enable a progressive transition from general linguistic knowledge to strong domain-specific expertise. Experimental results across diverse medical NLP tasks, including question answering, preference classification, and response generation, demonstrate that ICFT consistently outperforms state-of-the-art baselines, achieving improvements in both accuracy and efficiency. Further analysis reveals the framework's ability to generalize to unseen data, reduce errors, and deliver diverse, contextually relevant medical responses. These findings establish ICFT as a robust and scalable solution for adapting LLMs to the medical domain, offering practical benefits for real-world healthcare applications.