The Patient is not a Moving Document: A World Model Training Paradigm for Longitudinal EHR

TL;DR

The paper argues that traditional next-token predictive models treat patients as static documents rather than dynamic systems. It introduces SMB-Structure, a world-model framework that grounds clinical semantics with supervised fine-tuning (SFT) while forcing latent-space trajectory forecasting via Joint-Embedding Predictive Architecture (JEPA). By training with both objectives and employing a momentum encoder, SMB-Structure encodes patient state evolution and improves long-horizon predictions across two large cohorts (MSK and INSPECT) using linear probes along disease trajectories. The results highlight the importance of trajectory diversity and curriculum grounding to capture dynamic clinical trajectories, suggesting substantial potential for more accurate, dynamics-aware clinical AI. The work provides a foundation for future counterfactual and intervention-conditioned world models in longitudinal EHR analytics, with open-source model weights available for further exploration.

Abstract

Large language models (LLMs) trained with next-word-prediction have achieved success as clinical foundation models. Representations from these language backbones yield strong linear probe performance across biomedical tasks, suggesting that patient semantics emerge from next-token prediction at scale. However, this paradigm treats patients as a document to be summarized rather than a dynamical system to be simulated; a patient's trajectory emerges from their state evolving under interventions and time, requiring models that simulate dynamics rather than predict tokens. To address this, we introduce SMB-Structure, a world model for structured EHR that grounds a joint-embedding prediction architecture (JEPA) with next-token prediction (SFT). SFT grounds our model to reconstruct future patient states in token space, while JEPA predicts those futures in latent space from the initial patient representation alone, forcing trajectory dynamics to be encoded before the next state is observed. We validate across two large-scale cohorts: Memorial Sloan Kettering (23,319 oncology patients; 323,000+ patient-years) and INSPECT (19,402 pulmonary embolism patients). Using a linear probe evaluated at multiple points along the disease trajectory, we demonstrate that our training paradigm learns embeddings that capture disease dynamics not recoverable by autoregressive baselines, enabling SMB-Structure to achieve competitive performance on complex tasks characterized by high patient heterogeneity. Model weights are available at https://huggingface.co/standardmodelbio/SMB-v1-1.7B-Structure.

Figures (4)

• Figure 1: Architecture for SMB-Structure for Time-to-Event EHR Modeling.
• Figure 2: Evaluation Framework of Foundation Models for Time-to-Event EHR.
• Figure 3: Model performance by oncology indication on MSK cohort.
• Figure 4: Model performance benchmark on Lung Cancer.