Temporal Fusion Nexus: A task-agnostic multi-modal embedding model for clinical narratives and irregular time series in post-kidney transplant care

TL;DR

Temporal Fusion Nexus (TFN) presents a task-agnostic, multi-modal embedding that fuses irregular longitudinal data with unstructured clinical narratives to predict post-kidney transplant outcomes. It combines a time-series module (TM-LSTM) and a text module (Med-GTE-hybrid-de) through cross-attention, with a disentangled latent space enforced by decorrelation and sparsity-driven losses. In a retrospective cohort of 3382 kidney transplant recipients, TFN achieves AUCs of 0.96 for graft loss, 0.84 for graft rejection, and 0.86 for mortality, outperforming unimodal baselines and state-of-the-art CDSS while enabling interpretable attributions via SHAP and structured latent representations. The approach is shown to leverage multi-modal information additively, calibrate well after post-processing, and reveal temporally meaningful patterns; its task-agnostic design suggests broad applicability to other clinical domains with heterogeneous data modalities and irregular sampling.

Abstract

We introduce Temporal Fusion Nexus (TFN), a multi-modal and task-agnostic embedding model to integrate irregular time series and unstructured clinical narratives. We analysed TFN in post-kidney transplant (KTx) care, with a retrospective cohort of 3382 patients, on three key outcomes: graft loss, graft rejection, and mortality. Compared to state-of-the-art model in post KTx care, TFN achieved higher performance for graft loss (AUC 0.96 vs. 0.94) and graft rejection (AUC 0.84 vs. 0.74). In mortality prediction, TFN yielded an AUC of 0.86. TFN outperformed unimodal baselines (approx 10% AUC improvement over time series only baseline, approx 5% AUC improvement over time series with static patient data). Integrating clinical text improved performance across all tasks. Disentanglement metrics confirmed robust and interpretable latent factors in the embedding space, and SHAP-based attributions confirmed alignment with clinical reasoning. TFN has potential application in clinical tasks beyond KTx, where heterogeneous data sources, irregular longitudinal data, and rich narrative documentation are available.

Figures (9)

• Figure 1: Overview of the TFN modelling approach. (a) Modality specific modules, Med-GTE-hybrid-de and TM-LSTM, encode clinical texts and irregular time series data respectively. Static patient data is integrated with time series data through a variable-wise gating mechanism that selectively updates the TM-LSTM's hidden state. (b) Embeddings are generated for the text and time series modalities independently. The text embeddings are generated for each documented time point. The time series embeddings (with integrated static data) are also generated for each documented time point. A multi-head self-attention mechanism with a causal mask (temporal attention) is also utilised to capture the variable importance of different time points in patient's trajectory. (c) Cross-attention mechanism is used to combine the clinical text embeddings and time series embeddings. Queries (Q) are computed using the time series embeddings, while Keys (K) and Values (V) are obtained from linearly projected text embeddings. Interaction between the modalities are computed using scalar dot-product. (d) The resulting representation (attention output X) is a multi-modal shared embedding space, referred to as the Nexus. Downstream prediction performance is demonstrated on three key KTx outcomes: graft rejection, graft loss and mortality.
• Figure 2: Trajectory of a representative KTx recipient in our cohort. Horiz. axis shows a relative time in days after transplantation. Static features included age, gender, blood group. Longitudinally collected time series features included blood pressure, urine volume, heart rate, serum creatinine, etc. Medications and clinical notes, which involves discharge summaries, radiology reports, etc. were documented longitudinally. Two key clinical outcomes are shown in the patient trajectory: graft rejection (Day 163) and graft loss (Day 341), besides several episodes of hospital admissions.
• Figure 3: AUC of the multi-step prediction horizon during training of the time series encoder module TM-LSTM. Three clinical prediction tasks were considered: (a) Graft loss, (b) Graft rejection, (c) Mortality. Four prediction windows (30 days, 90 days, 180 days, and 360 days) were analysed (see Sec. \ref{['2.2']} for details). AUC for all three tasks peaked around 10 steps, which marks the empirically optimal context to capture clinically relevant temporal patterns.
• Figure 4: Performance comparison between our fine-tuned text encoder module Med-GTE-hybrid-de and the base model, gte-large. AUC scores are plotted against four prediction windows (30, 90, 180 and 360 days) across three prediction tasks: (a) Graft loss (b) Graft rejection (c) Mortality. Error bars represent standard deviation across five-fold cross-validations. The Med-GTE-hybrid-de model consistently outperformed GTE-large across all tasks and prediction windows, with the most notable improvement observed in graft loss prediction.
• Figure 5: Illustration of additive effect of multi-modal data integration across three clinical prediction tasks: (a) Graft loss (b) Graft rejection (c) Mortality. Four prediction windows (30, 90, 180, 360 days) were analysed. We compare AUC of TFN model, which uses all three data modalities (i.e., time series, static patient data and clinical notes) to a time series only baseline, and a time series with static patient data baseline. Error bars represent standard deviation across five-fold cross-validations. The consistent improvements in performance for TFN with all modalities demonstrate the additive value of each modality, and confirms that the TFN model could harness complementary information from the different modalities.