Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Building the EHR Foundation Model via Next Event Prediction

Zekai Chen, Arda Pekis, Kevin Brown

TL;DR

This work introduces Next Event Prediction (NEP), a framework that fine-tunes large language models on timestamped sequences of clinical events to explicitly model temporal dynamics and causal relationships in EHR data. By predicting the next clinical event autoregressively and then using the resulting embeddings for downstream tasks, NEP achieves state-of-the-art or competitive results on oncology survival prediction and diverse clinical tasks, while demonstrating strong data efficiency. The approach yields interpretable attention patterns aligned with known disease pathways and remains compatible with existing EHR encoders, offering a scalable path toward temporally aware clinical AI. The findings suggest NEP can generalize beyond disease targets and reduce labeling requirements, with potential impact for proactive, longitudinal patient care.

Abstract

Electronic Health Records (EHRs) contain rich temporal dynamics that conventional encoding approaches fail to adequately capture. While Large Language Models (LLMs) show promise for EHR modeling, they struggle to reason about sequential clinical events and temporal dependencies. We propose Next Event Prediction (NEP), a framework that enhances LLMs' temporal reasoning through autoregressive fine-tuning on clinical event sequences. By reformulating EHRs as timestamped event chains and predicting future medical events, NEP explicitly models disease progression patterns and causal relationships. Extensive evaluations across oncology survival prediction and clinical diagnosis tasks demonstrate NEP's superiority, outperforming specialized EHR models by 4.6% AUROC and general-purpose LLMs by 7.2% C-index in temporal reasoning tasks. Our analyses reveal dual benefits: state-of-the-art prediction accuracy combined with clinically interpretable attention patterns that align with known disease pathways.

Building the EHR Foundation Model via Next Event Prediction

TL;DR

This work introduces Next Event Prediction (NEP), a framework that fine-tunes large language models on timestamped sequences of clinical events to explicitly model temporal dynamics and causal relationships in EHR data. By predicting the next clinical event autoregressively and then using the resulting embeddings for downstream tasks, NEP achieves state-of-the-art or competitive results on oncology survival prediction and diverse clinical tasks, while demonstrating strong data efficiency. The approach yields interpretable attention patterns aligned with known disease pathways and remains compatible with existing EHR encoders, offering a scalable path toward temporally aware clinical AI. The findings suggest NEP can generalize beyond disease targets and reduce labeling requirements, with potential impact for proactive, longitudinal patient care.

Abstract

Electronic Health Records (EHRs) contain rich temporal dynamics that conventional encoding approaches fail to adequately capture. While Large Language Models (LLMs) show promise for EHR modeling, they struggle to reason about sequential clinical events and temporal dependencies. We propose Next Event Prediction (NEP), a framework that enhances LLMs' temporal reasoning through autoregressive fine-tuning on clinical event sequences. By reformulating EHRs as timestamped event chains and predicting future medical events, NEP explicitly models disease progression patterns and causal relationships. Extensive evaluations across oncology survival prediction and clinical diagnosis tasks demonstrate NEP's superiority, outperforming specialized EHR models by 4.6% AUROC and general-purpose LLMs by 7.2% C-index in temporal reasoning tasks. Our analyses reveal dual benefits: state-of-the-art prediction accuracy combined with clinically interpretable attention patterns that align with known disease pathways.

Paper Structure

This paper contains 19 sections, 3 equations, 1 figure, 13 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Next Event Prediction
  4. EHR as a Sequence of Events
  5. Causal Mask vs. Bi-directional
  6. Implementation Details
  7. Data Sampling.
  8. Training paradigm.
  9. Implementations.
  10. Experiments
  11. Datasets and Metrics
  12. MSK-CHORD
  13. EHRSHOT
  14. Baselines
  15. Results & Analysis
  16. ...and 4 more sections

Figures (1)

  • Figure 1: Overview of the Next Event Prediction (NEP) framework. Patient clinical histories are serialized into timestamped sequences of events and formatted into structured prompts, separated by comprehensive and diverse event types. Large Language Models (LLMs) are fine-tuned to autoregressively predict subsequent clinical events, explicitly capturing temporal and causal dependencies within patient trajectories. Embeddings from the fine-tuned LLM are subsequently utilized to perform downstream clinical prediction tasks using lightweight classification heads (e.g., logistic regression).