Limits of Generative Pre-Training in Structured EMR Trajectories with Irregular Sampling

TL;DR

The study probes whether autoregressive generative pre-training on irregular, structured EMR trajectories yields clinically meaningful representations. It trains a sequence-to-sequence LSTM and a reduced ETHOS-style Transformer on two longitudinal cohorts (ART for HIV and Acute Hypotension) with controlled irregular sampling and a time-gap encoding, evaluating forward extrapolation via distributional and correlational metrics. Findings show generated trajectories preserve marginal distributions but fail to maintain cross-feature correlations, especially under larger sampling gaps, suggesting limited clinical coherence. The work emphasizes the necessity of domain-specific validation and proposes patient-trajectory synthesis as a practical probe to assess representation fidelity before downstream fine-tuning or deployment.

Abstract

Foundation models refer to architectures trained on vast datasets using autoregressive pre-training from natural language processing to capture intricate patterns and motifs. They were originally developed to transfer such learned knowledge to downstream predictive tasks. Recently, however, some studies repurpose these learned representations for phenotype discovery without rigorous validation, risking superficially realistic but clinically incoherent embeddings. To test this mismatch, we trained two autoregressive models -- a sequence-to-sequence LSTM and a reduced Transformer -- on longitudinal ART for HIV and Acute Hypotension datasets. Controlled irregularity was added during training via random inter-visit gaps, while test sequences stayed complete. Patient-trajectory synthesis evaluated distributional and correlational fidelity. Both reproduced feature distributions but failed to preserve cross-feature structure -- showing that generative pre-training yields local realism but limited clinical coherence. These results highlight the need for domain-specific evaluation and support trajectory synthesis as a practical probe before fine-tuning or deployment.

Figures (3)

• Figure 1: Demonstrating the dual-split design using the ART for HIV dataset.
• Figure 2: Distributional fidelity: ART for HIV (rows 1–2) and Acute Hypotension (rows 3–4).
• Figure 3: Correlational fidelity: ART for HIV (row 1) and Acute Hypotension (row 2).