Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Do We Still Need Clinical Language Models?

Eric Lehman, Evan Hernandez, Diwakar Mahajan, Jonas Wulff, Micah J. Smith, Zachary Ziegler, Daniel Nadler, Peter Szolovits, Alistair Johnson, Emily Alsentzer

TL;DR

The paper investigates whether specialized clinical language models remain valuable given advances in general-domain LLMs. Through an extensive evaluation of 12 models across three EHR-derived tasks (MedNLI, RadQA, CLIP) and training Clinical-T5 variants from MIMIC data, it demonstrates that small, in-domain models substantially outperform in-context learning baselines and can match or exceed larger general-domain LMs. A key finding is that pretraining on in-domain clinical tokens yields parameter-efficient models, underscoring the importance of domain alignment over sheer scale. The authors conclude that specialized clinical modeling remains essential for safety-critical healthcare settings and provide a cost-focused argument—via FLOPs and token budgets—for deploying compact clinical models in real-world hospitals.

Abstract

Although recent advances in scaling large language models (LLMs) have resulted in improvements on many NLP tasks, it remains unclear whether these models trained primarily with general web text are the right tool in highly specialized, safety critical domains such as clinical text. Recent results have suggested that LLMs encode a surprising amount of medical knowledge. This raises an important question regarding the utility of smaller domain-specific language models. With the success of general-domain LLMs, is there still a need for specialized clinical models? To investigate this question, we conduct an extensive empirical analysis of 12 language models, ranging from 220M to 175B parameters, measuring their performance on 3 different clinical tasks that test their ability to parse and reason over electronic health records. As part of our experiments, we train T5-Base and T5-Large models from scratch on clinical notes from MIMIC III and IV to directly investigate the efficiency of clinical tokens. We show that relatively small specialized clinical models substantially outperform all in-context learning approaches, even when finetuned on limited annotated data. Further, we find that pretraining on clinical tokens allows for smaller, more parameter-efficient models that either match or outperform much larger language models trained on general text. We release the code and the models used under the PhysioNet Credentialed Health Data license and data use agreement.

Do We Still Need Clinical Language Models?

TL;DR

The paper investigates whether specialized clinical language models remain valuable given advances in general-domain LLMs. Through an extensive evaluation of 12 models across three EHR-derived tasks (MedNLI, RadQA, CLIP) and training Clinical-T5 variants from MIMIC data, it demonstrates that small, in-domain models substantially outperform in-context learning baselines and can match or exceed larger general-domain LMs. A key finding is that pretraining on in-domain clinical tokens yields parameter-efficient models, underscoring the importance of domain alignment over sheer scale. The authors conclude that specialized clinical modeling remains essential for safety-critical healthcare settings and provide a cost-focused argument—via FLOPs and token budgets—for deploying compact clinical models in real-world hospitals.

Abstract

Although recent advances in scaling large language models (LLMs) have resulted in improvements on many NLP tasks, it remains unclear whether these models trained primarily with general web text are the right tool in highly specialized, safety critical domains such as clinical text. Recent results have suggested that LLMs encode a surprising amount of medical knowledge. This raises an important question regarding the utility of smaller domain-specific language models. With the success of general-domain LLMs, is there still a need for specialized clinical models? To investigate this question, we conduct an extensive empirical analysis of 12 language models, ranging from 220M to 175B parameters, measuring their performance on 3 different clinical tasks that test their ability to parse and reason over electronic health records. As part of our experiments, we train T5-Base and T5-Large models from scratch on clinical notes from MIMIC III and IV to directly investigate the efficiency of clinical tokens. We show that relatively small specialized clinical models substantially outperform all in-context learning approaches, even when finetuned on limited annotated data. Further, we find that pretraining on clinical tokens allows for smaller, more parameter-efficient models that either match or outperform much larger language models trained on general text. We release the code and the models used under the PhysioNet Credentialed Health Data license and data use agreement.
Paper Structure (28 sections, 4 equations, 4 figures, 15 tables)

This paper contains 28 sections, 4 equations, 4 figures, 15 tables.

Table of Contents

  1. Introduction
  2. Background & Related Work
  3. Specialized Clinical Models
  4. Finetuning General Purpose LLMs for Clinical Tasks
  5. Using In-Context Learning
  6. Experimental Setup
  7. Tasks
  8. Models
  9. Clinical Models Are Parameter Efficient
  10. When Is Pretraining From Scratch More Efficient?
  11. In-Domain Tokens Are More Valuable
  12. In-Context Learning Underperforms Task Specific Models
  13. Limitations & Future Work
  14. Conclusion
  15. Acknowledgments
  16. ...and 13 more sections

Figures (4)

  • Figure 1: We consider three options for how a healthcare system with access to clinical notes might approach a clinical problem. First, the healthcare system could use a specialized language model pretrained on clinical notes. This model could be pretrained from scratch (Row 1) or from a publicly available checkpoint of a LM pretrained on general text (Row 2). Alternatively, the healthcare system could directly finetune a publicly available general-purpose language model to perform the clinical task (Row 3). Finally, the healthcare system could use a state-of-the-art LLM such as GPT-3, without any additional finetuning, by prompting the LLM to perform the clinical task (Row 4).
  • Figure 2: An example of the tasks we consider in this paper. In MedNLI, the goal is determine if the two sentences entail, contradict or are neutral to each other. RadQA is an extractive question answering task over radiology reports. In CLIP, the goal is to identify the different types of patient follow-up information in each sentence of a discharge summary (if any). These examples illustrate the difficulty of parsing clinical text.
  • Figure 3: Log total pretraining FLOPs by performance for MedNLI, RadQA, and CLIP. When comparing models with a similar number of FLOPs or performance, clinical models outperform general models. We add regression curves for all T5 models, which are comparable in architecture and training process and differ only in model size and pretraining domain. The T5 models demonstrate the effectiveness of clinical tokens relative to tokens taken from the general web.
  • Figure 4: An ablation study in which we compare models trained with 1%, 5%, 10%, 25%, and 100% of available training data for each task. Except for RadQA at 1%, GPT-3 and T5-Flan-XXL perform worse than GatorTron at all ablation points. We report mean performance over three random seeds.