Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Serialized EHR make for good text representations

Zhirong Chou, Quan Qin, Shi Li

TL;DR

SerialBEHRT introduces a domain-aligned foundation model that pretrains on a joint corpus of scientific text and serialized Electronic Health Records (EHR), using a masked language modeling objective $L_{MLM}(\theta)$ to learn temporal and contextual relationships among clinical events. By expanding SciBERT's vocabulary to include $T_{Serialized_EHR}$ tokens and transforming tabular EHR data into text via serialization, the model achieves richer patient representations and improves antibiotic susceptibility prediction on MIMIC-IV data. Across foundation-model baselines and representation strategies, SerialBEHRT demonstrates superior and stable performance, underscoring the value of temporal serialization and modality-aligned pretraining for healthcare tasks. The work also shows that a single architecture can coherently encode two disparate data sources, enabling more versatile and clinically actionable embeddings for downstream decision support.

Abstract

The emergence of foundation models in healthcare has opened new avenues for learning generalizable representations from large scale clinical data. Yet, existing approaches often struggle to reconcile the tabular and event based nature of Electronic Health Records (EHRs) with the sequential priors of natural language models. This structural mismatch limits their ability to capture longitudinal dependencies across patient encounters. We introduce SerialBEHRT, a domain aligned foundation model that extends SciBERT through additional pretraining on structured EHR sequences. SerialBEHRT is designed to encode temporal and contextual relationships among clinical events, thereby producing richer patient representations. We evaluate its effectiveness on the task of antibiotic susceptibility prediction, a clinically meaningful problem in antibiotic stewardship. Through extensive benchmarking against state of the art EHR representation strategies, we demonstrate that SerialBEHRT achieves superior and more consistent performance, highlighting the importance of temporal serialization in foundation model pretraining for healthcare.

Serialized EHR make for good text representations

TL;DR

SerialBEHRT introduces a domain-aligned foundation model that pretrains on a joint corpus of scientific text and serialized Electronic Health Records (EHR), using a masked language modeling objective to learn temporal and contextual relationships among clinical events. By expanding SciBERT's vocabulary to include tokens and transforming tabular EHR data into text via serialization, the model achieves richer patient representations and improves antibiotic susceptibility prediction on MIMIC-IV data. Across foundation-model baselines and representation strategies, SerialBEHRT demonstrates superior and stable performance, underscoring the value of temporal serialization and modality-aligned pretraining for healthcare tasks. The work also shows that a single architecture can coherently encode two disparate data sources, enabling more versatile and clinically actionable embeddings for downstream decision support.

Abstract

The emergence of foundation models in healthcare has opened new avenues for learning generalizable representations from large scale clinical data. Yet, existing approaches often struggle to reconcile the tabular and event based nature of Electronic Health Records (EHRs) with the sequential priors of natural language models. This structural mismatch limits their ability to capture longitudinal dependencies across patient encounters. We introduce SerialBEHRT, a domain aligned foundation model that extends SciBERT through additional pretraining on structured EHR sequences. SerialBEHRT is designed to encode temporal and contextual relationships among clinical events, thereby producing richer patient representations. We evaluate its effectiveness on the task of antibiotic susceptibility prediction, a clinically meaningful problem in antibiotic stewardship. Through extensive benchmarking against state of the art EHR representation strategies, we demonstrate that SerialBEHRT achieves superior and more consistent performance, highlighting the importance of temporal serialization in foundation model pretraining for healthcare.
Paper Structure (32 sections, 3 equations, 1 figure, 4 tables)

This paper contains 32 sections, 3 equations, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Medical Representation Learning
  4. Biomedical Language Models
  5. Text Serialization
  6. Methods
  7. SerialBEHRT
  8. Data Coverage
  9. Vocabulary Expansion and Tokenizer Adaptation
  10. Masked Language Modeling (MLM) for SerialBEHRT
  11. Pre-training Optimizations
  12. Data and Cohort Selection
  13. Pre-training Data
  14. Antibiotics Study
  15. Benchmarking
  16. ...and 17 more sections

Figures (1)

  • Figure 1: Schematic representation of SerialBEHRT, pre-trained on a composite of scientific texts and serialized EHRs, demonstrating the Masked Language Modeling (MLM) objective.