Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Rephrasing Electronic Health Records for Pretraining Clinical Language Models

Jinghui Liu, Anthony Nguyen

TL;DR

This work tackles privacy constraints that limit access to large-scale EHR data by proposing a rephrasing approach: real clinical notes are transformed into synthetic pretraining corpora using small, instruction-tuned LLMs. By employing medically adapted prompts across four sub-10B models and real discharge summaries from MIMIC data, the authors generate 20M-token synthetic datasets and evaluate their impact on both decoder- and encoder-based language models. Perplexity-based evaluations show that rephrasing outperforms prior synthetic approaches, and combining synthetic with real notes further enhances downstream performance on clinical NLP tasks, sometimes surpassing the established ClinicalBERT baseline. The findings suggest a practical pathway for institution-scale pretraining with privacy-preserving synthetic data, enabling scalable development of high-quality clinical language models while mitigating data-sharing constraints.

Abstract

Clinical language models are important for many applications in healthcare, but their development depends on access to extensive clinical text for pretraining. However, obtaining clinical notes from electronic health records (EHRs) at scale is challenging due to patient privacy concerns. In this study, we rephrase existing clinical notes using LLMs to generate synthetic pretraining corpora, drawing inspiration from previous work on rephrasing web data. We examine four popular small-sized LLMs (<10B) to create synthetic clinical text to pretrain both decoder-based and encoder-based language models. The method yields better results in language modeling and downstream tasks than previous synthesis approaches without referencing real clinical text. We find that augmenting original clinical notes with synthetic corpora from different LLMs improves performances even at a small token budget, showing the potential of this method to support pretraining at the institutional level or be scaled to synthesize large-scale clinical corpora.

Rephrasing Electronic Health Records for Pretraining Clinical Language Models

TL;DR

This work tackles privacy constraints that limit access to large-scale EHR data by proposing a rephrasing approach: real clinical notes are transformed into synthetic pretraining corpora using small, instruction-tuned LLMs. By employing medically adapted prompts across four sub-10B models and real discharge summaries from MIMIC data, the authors generate 20M-token synthetic datasets and evaluate their impact on both decoder- and encoder-based language models. Perplexity-based evaluations show that rephrasing outperforms prior synthetic approaches, and combining synthetic with real notes further enhances downstream performance on clinical NLP tasks, sometimes surpassing the established ClinicalBERT baseline. The findings suggest a practical pathway for institution-scale pretraining with privacy-preserving synthetic data, enabling scalable development of high-quality clinical language models while mitigating data-sharing constraints.

Abstract

Clinical language models are important for many applications in healthcare, but their development depends on access to extensive clinical text for pretraining. However, obtaining clinical notes from electronic health records (EHRs) at scale is challenging due to patient privacy concerns. In this study, we rephrase existing clinical notes using LLMs to generate synthetic pretraining corpora, drawing inspiration from previous work on rephrasing web data. We examine four popular small-sized LLMs (<10B) to create synthetic clinical text to pretrain both decoder-based and encoder-based language models. The method yields better results in language modeling and downstream tasks than previous synthesis approaches without referencing real clinical text. We find that augmenting original clinical notes with synthetic corpora from different LLMs improves performances even at a small token budget, showing the potential of this method to support pretraining at the institutional level or be scaled to synthesize large-scale clinical corpora.

Paper Structure

This paper contains 14 sections, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Rephrasing Clinical Notes with LLMs
  3. Medically Adapted Prompts
  4. LLMs for Rephrasing
  5. Source Clinical Notes
  6. Perplexity Evaluation with Causal Language Models
  7. Experimental Setup
  8. Results
  9. Downstream Evaluation with Masked Language Models
  10. Experimental Setup
  11. Results
  12. Discussion
  13. Conclusion
  14. Examples of Rephrased Notes

Figures (3)

  • Figure 1: Perplexity scores of language models pretrained on different synthetic sources. Ascplepius refers the synthetic notes from Kweon2024-xk. The four LLMs refer to their synthetic corpora based on the rephrasing method, respectively. Lower perplexity means better language modeling performances.
  • Figure 2: Perplexity scores of language models pretrained on real and synthetic notes. Higher red dashed line indicates the performance with real notes alone.
  • Figure 3: Augmentation performance with synthetic data using different prompts.