Biomedical Large Languages Models Seem not to be Superior to Generalist Models on Unseen Medical Data

TL;DR

The paper investigates whether biomedical fine-tuning of large language models yields real gains on unseen clinical data. By benchmarking biomedical LLMs against strong generalist baselines across NEJM/JAMA case challenges and a suite of CLUE benchmarks, it reveals that generalist models often outperform domain-specific ones, especially on non-medical tasks, with larger models narrowing the gap. The authors discuss potential explanations, including broader exposure during pretraining and the risks of catastrophic forgetting, and highlight retrieval-augmented generation as a promising alternative. The work underscores the need for rigorous, task-specific evaluation in healthcare AI and cautions against assuming universal benefits from domain-focused fine-tuning.

Abstract

Large language models (LLMs) have shown potential in biomedical applications, leading to efforts to fine-tune them on domain-specific data. However, the effectiveness of this approach remains unclear. This study evaluates the performance of biomedically fine-tuned LLMs against their general-purpose counterparts on a variety of clinical tasks. We evaluated their performance on clinical case challenges from the New England Journal of Medicine (NEJM) and the Journal of the American Medical Association (JAMA) and on several clinical tasks (e.g., information extraction, document summarization, and clinical coding). Using benchmarks specifically chosen to be likely outside the fine-tuning datasets of biomedical models, we found that biomedical LLMs mostly perform inferior to their general-purpose counterparts, especially on tasks not focused on medical knowledge. While larger models showed similar performance on case tasks (e.g., OpenBioLLM-70B: 66.4% vs. Llama-3-70B-Instruct: 65% on JAMA cases), smaller biomedical models showed more pronounced underperformance (e.g., OpenBioLLM-8B: 30% vs. Llama-3-8B-Instruct: 64.3% on NEJM cases). Similar trends were observed across the CLUE (Clinical Language Understanding Evaluation) benchmark tasks, with general-purpose models often performing better on text generation, question answering, and coding tasks. Our results suggest that fine-tuning LLMs to biomedical data may not provide the expected benefits and may potentially lead to reduced performance, challenging prevailing assumptions about domain-specific adaptation of LLMs and highlighting the need for more rigorous evaluation frameworks in healthcare AI. Alternative approaches, such as retrieval-augmented generation, may be more effective in enhancing the biomedical capabilities of LLMs without compromising their general knowledge.

Figures (1)

• Figure 1: Comparative performance of the domain specific and general-purpose base models on the two clinical case vignette series. Bars represent the respective metric and error bars represent 95% confidence intervals obtained through bootstrapping.