Clinical ModernBERT: An efficient and long context encoder for biomedical text

TL;DR

Clinical ModernBERT presents a domain-adapted, long-context encoder for biomedical text that merges ModernBERT’s architectural advances with biomedical pretraining on PubMed, MIMIC-IV, and structured ontologies. Key innovations include RoPE for relative positional encoding, GeGLU activations, bias-free layers, Flash Attention, and an 8,192-token context, enabling richer representations for long clinical documents. Empirical results show competitive to state-of-the-art performance across short- and long-context clinical NLP benchmarks, with strong improvements in medical-code alignment and retrieval tasks, plus superior efficiency at scale. The work demonstrates how targeted domain data, ontology integration, and architectural modernization yield robust, scalable clinical NLP encoders suitable for high-stakes healthcare applications.

Abstract

We introduce Clinical ModernBERT, a transformer based encoder pretrained on large scale biomedical literature, clinical notes, and medical ontologies, incorporating PubMed abstracts, MIMIC IV clinical data, and medical codes with their textual descriptions. Building on ModernBERT the current state of the art natural language text encoder featuring architectural upgrades such as rotary positional embeddings (RoPE), Flash Attention, and extended context length up to 8,192 tokens our model adapts these innovations specifically for biomedical and clinical domains. Clinical ModernBERT excels at producing semantically rich representations tailored for long context tasks. We validate this both by analyzing its pretrained weights and through empirical evaluation on a comprehensive suite of clinical NLP benchmarks.

Figures (4)

• Figure 1: Medical Code Ontologies Construction: An illustration of structured ontology construction across multiple ICD code versions. Each row represents a distinct medical concept identified by its version-specific code and description, which is then converted into a standardized, descriptive natural language representation. This process facilitates alignment and interoperability across evolving coding schemes. This setup is inspired by methods like hegselmann2023tabllmono2024text which use text templates to serialize tabular data.
• Figure 2: ICD-9 tSNE Latent Space Visualization: A tSNE visualization of the ICD 9 Diagnoses codes using modernBERT versus Clinical ModernBERT. This visualization provides the added use of adding the medical code ontologies as a pre-training source to encode coded language seen frequently in clinical practice.
• Figure 3: Comparative Performance Analysis of BERT Models: This figure demonstrates the processing time requirements across three BERT variants (Distil-BERT, BioClinicalBERT, and Clinical ModernBERT) as data volume increases from 10,000 to 100,000 points. BioClinicalBERT consistently shows the highest computational demand, requiring approximately 1.4x the processing time of Distil-BERT and 1.6x that of Clinical ModernBERT at maximum load. Clinical ModernBERT demonstrates superior efficiency, maintaining the lowest processing times across all data volumes, making it optimal for resource-constrained environments.
• Figure 4: Masked Language Modeling (MLM) Top-K Accuracies and Loss: We report top-K accuracies for $k = 1, 5, 10, 25$ alongside MLM loss across three pre-training runs initialized with different learning rates (top to bottom: $3 \times 10^{-3}$, $5 \times 10^{-4}$, $1 \times 10^{-5}$). Higher learning rates yielded more stable convergence and avoided shallow local minima, suggesting improved exploration of the loss landscape. As expected, larger learning rates also introduced noisier gradient updates, which aligns with standard intuitions in stochastic optimization.