Learning Longitudinal Health Representations from EHR and Wearable Data

TL;DR

This work introduces a multimodal health foundation model that jointly pretrains on irregular EHR data and dense wearable signals to learn a continuous-time latent health state $x_i(t)$. By combining modality-specific encoders with a shared continuous-time backbone and cross-modal self-supervised objectives, the model achieves temporally coherent, clinically grounded representations. Across clinical event forecasting, physiological state estimation, and longitudinal risk modeling, it outperforms unimodal and late-fusion baselines, with notable gains at long horizons and under missing data, while improving calibration and interpretability. The approach demonstrates the value of cross-modal pretraining for faithful longitudinal health representations and lays groundwork for robust, privacy-conscious deployment with attention to biases and distribution shifts.

Abstract

Foundation models trained on electronic health records show strong performance on many clinical prediction tasks but are limited by sparse and irregular documentation. Wearable devices provide dense continuous physiological signals but lack semantic grounding. Existing methods usually model these data sources separately or combine them through late fusion. We propose a multimodal foundation model that jointly represents electronic health records and wearable data as a continuous time latent process. The model uses modality specific encoders and a shared temporal backbone pretrained with self supervised and cross modal objectives. This design produces representations that are temporally coherent and clinically grounded. Across forecasting physiological and risk modeling tasks the model outperforms strong electronic health record only and wearable only baselines especially at long horizons and under missing data. These results show that joint electronic health record and wearable pretraining yields more faithful representations of longitudinal health.