Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

UniPACT: A Multimodal Framework for Prognostic Question Answering on Raw ECG and Structured EHR

Jialu Tang, Tong Xia, Yuan Lu, Aaqib Saeed

TL;DR

UniPACT tackles the problem of enabling LLM-based prognosis over heterogeneous clinical data by introducing a structured EHR prompting scheme and a dedicated ECG waveform encoder to fuse raw ECG with structured EHR in a single generative model. The approach unifies multimodal representations through an MM-Projector and a MedGemma-4B/LLaVA-based backbone, trained with a two-stage LoRA fine-tuning regime under a single multi-task objective. It achieves state-of-the-art performance on the MDS-ED benchmark, with a mean AUROC of $89.37\%$, and demonstrates robustness to missing data and cross-task generalization across diagnosis, deterioration, ICU admission, and mortality. These results highlight the practical potential of end-to-end multimodal prognostic reasoning in clinical settings, enabling a single model to answer diverse prognostic questions with high fidelity.

Abstract

Accurate clinical prognosis requires synthesizing structured Electronic Health Records (EHRs) with real-time physiological signals like the Electrocardiogram (ECG). Large Language Models (LLMs) offer a powerful reasoning engine for this task but struggle to natively process these heterogeneous, non-textual data types. To address this, we propose UniPACT (Unified Prognostic Question Answering for Clinical Time-series), a unified framework for prognostic question answering that bridges this modality gap. UniPACT's core contribution is a structured prompting mechanism that converts numerical EHR data into semantically rich text. This textualized patient context is then fused with representations learned directly from raw ECG waveforms, enabling an LLM to reason over both modalities holistically. We evaluate UniPACT on the comprehensive MDS-ED benchmark, it achieves a state-of-the-art mean AUROC of 89.37% across a diverse set of prognostic tasks including diagnosis, deterioration, ICU admission, and mortality, outperforming specialized baselines. Further analysis demonstrates that our multimodal, multi-task approach is critical for performance and provides robustness in missing data scenarios.

UniPACT: A Multimodal Framework for Prognostic Question Answering on Raw ECG and Structured EHR

TL;DR

UniPACT tackles the problem of enabling LLM-based prognosis over heterogeneous clinical data by introducing a structured EHR prompting scheme and a dedicated ECG waveform encoder to fuse raw ECG with structured EHR in a single generative model. The approach unifies multimodal representations through an MM-Projector and a MedGemma-4B/LLaVA-based backbone, trained with a two-stage LoRA fine-tuning regime under a single multi-task objective. It achieves state-of-the-art performance on the MDS-ED benchmark, with a mean AUROC of , and demonstrates robustness to missing data and cross-task generalization across diagnosis, deterioration, ICU admission, and mortality. These results highlight the practical potential of end-to-end multimodal prognostic reasoning in clinical settings, enabling a single model to answer diverse prognostic questions with high fidelity.

Abstract

Accurate clinical prognosis requires synthesizing structured Electronic Health Records (EHRs) with real-time physiological signals like the Electrocardiogram (ECG). Large Language Models (LLMs) offer a powerful reasoning engine for this task but struggle to natively process these heterogeneous, non-textual data types. To address this, we propose UniPACT (Unified Prognostic Question Answering for Clinical Time-series), a unified framework for prognostic question answering that bridges this modality gap. UniPACT's core contribution is a structured prompting mechanism that converts numerical EHR data into semantically rich text. This textualized patient context is then fused with representations learned directly from raw ECG waveforms, enabling an LLM to reason over both modalities holistically. We evaluate UniPACT on the comprehensive MDS-ED benchmark, it achieves a state-of-the-art mean AUROC of 89.37% across a diverse set of prognostic tasks including diagnosis, deterioration, ICU admission, and mortality, outperforming specialized baselines. Further analysis demonstrates that our multimodal, multi-task approach is critical for performance and provides robustness in missing data scenarios.
Paper Structure (8 sections, 4 equations, 1 figure, 3 tables)

This paper contains 8 sections, 4 equations, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Modality-Specific Representation Learning
  4. Multimodal Fusion and Generation
  5. Multi-Task Learning via Unified Prompting
  6. Results
  7. Conclusion
  8. Acknowledgments and Ethical Compliance

Figures (1)

  • Figure 1: The UniPACT framework for multimodal prognostic question answering. (a) Structured Prompt Formulation: Heterogeneous patient data, including structured EHR (demographics, biometrics, vitals) and a reference to the ECG waveform, are converted into a unified natural language prompt. This process transforms numerical values into a format that is natively understandable by the LLM. (b) Multimodal Fusion Architecture: A pretrained encoder processes the raw 12-lead ECG waveform to produce a feature embedding. A multimodal projector (MM-Projector) then aligns this ECG embedding with the LLM's text embedding space. These aligned ECG features are seamlessly integrated with the tokenized text prompt and processed by the LLM decoder for unified reasoning. (c) Prognostic Output Generation: The model generates a direct answer to the prognostic question (e.g., 'Yes'/'No' to a query about clinical deterioration).