Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LeMoF: Level-guided Multimodal Fusion for Heterogeneous Clinical Data

Jongseok Kim, Seongae Kang, Jonghwan Shin, Yuhan Lee, Ohyun Jo

TL;DR

LeMoF tackles the challenge of fusing heterogeneous ICU data for LOS prediction by exploiting level-wise representations within each modality. It introduces modality-aware level stacking, SHAP-based level importance, cross-modal attention, and stacked meta-predictions to produce a final estimate, all while selectively leveraging the most informative representation levels. Across MIMIC-IV and MIMIC-IV-ECG, LeMoF consistently outperforms state-of-the-art fusion baselines and demonstrates robustness across diseases. The study suggests that selective, level-guided fusion can generalize to diverse multimodal medical tasks, balancing stability and discriminative power in complex clinical settings.

Abstract

Multimodal clinical prediction is widely used to integrate heterogeneous data such as Electronic Health Records (EHR) and biosignals. However, existing methods tend to rely on static modality integration schemes and simple fusion strategies. As a result, they fail to fully exploit modality-specific representations. In this paper, we propose Level-guided Modal Fusion (LeMoF), a novel framework that selectively integrates level-guided representations within each modality. Each level refers to a representation extracted from a different layer of the encoder. LeMoF explicitly separates and learns global modality-level predictions from level-specific discriminative representations. This design enables LeMoF to achieve a balanced performance between prediction stability and discriminative capability even in heterogeneous clinical environments. Experiments on length of stay prediction using Intensive Care Unit (ICU) data demonstrate that LeMoF consistently outperforms existing state-of-the-art multimodal fusion techniques across various encoder configurations. We also confirmed that level-wise integration is a key factor in achieving robust predictive performance across various clinical conditions.

LeMoF: Level-guided Multimodal Fusion for Heterogeneous Clinical Data

TL;DR

LeMoF tackles the challenge of fusing heterogeneous ICU data for LOS prediction by exploiting level-wise representations within each modality. It introduces modality-aware level stacking, SHAP-based level importance, cross-modal attention, and stacked meta-predictions to produce a final estimate, all while selectively leveraging the most informative representation levels. Across MIMIC-IV and MIMIC-IV-ECG, LeMoF consistently outperforms state-of-the-art fusion baselines and demonstrates robustness across diseases. The study suggests that selective, level-guided fusion can generalize to diverse multimodal medical tasks, balancing stability and discriminative power in complex clinical settings.

Abstract

Multimodal clinical prediction is widely used to integrate heterogeneous data such as Electronic Health Records (EHR) and biosignals. However, existing methods tend to rely on static modality integration schemes and simple fusion strategies. As a result, they fail to fully exploit modality-specific representations. In this paper, we propose Level-guided Modal Fusion (LeMoF), a novel framework that selectively integrates level-guided representations within each modality. Each level refers to a representation extracted from a different layer of the encoder. LeMoF explicitly separates and learns global modality-level predictions from level-specific discriminative representations. This design enables LeMoF to achieve a balanced performance between prediction stability and discriminative capability even in heterogeneous clinical environments. Experiments on length of stay prediction using Intensive Care Unit (ICU) data demonstrate that LeMoF consistently outperforms existing state-of-the-art multimodal fusion techniques across various encoder configurations. We also confirmed that level-wise integration is a key factor in achieving robust predictive performance across various clinical conditions.
Paper Structure (22 sections, 13 equations, 3 figures, 3 tables)

This paper contains 22 sections, 13 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Challenge 1 : A Tendency Toward Modality-Specific Dependence.
  3. Challenge 2 : The Necessity of Selective Integration in Multimodal Healthcare.
  4. Related Work
  5. Multi-modal learning in medical domain
  6. Length of Stay prediction
  7. LeMoF : Level-guided Modal Fusion
  8. Modality-aware Level Stacking
  9. Stage 1 : Modality-specific Training
  10. Stage 2 : Level Importance Extraction.
  11. Stage 3 : Best-level Aggregation
  12. Level-guided Multimodal Fusion
  13. Integrated Decision Modeling
  14. Experiment
  15. Experimental Setting
  16. ...and 7 more sections

Figures (3)

  • Figure 1: LeMoF framework. Hierarchical representations from different encoder stages are selectively integrated via level-aware fusion.
  • Figure 2: Detailed illustration of the proposed LeMoF framework. Hierarchical representations are extracted from each modality using a Pyramid Feature Network, followed by level-wise prediction learning and importance-based best-level selection (Module 1 $\rightarrow$ M1). The selected representations are then used for level-guided cross-modal attention (Module 2 $\rightarrow$ M2) and integrated with modality-internal predictions to produce the final output (Module 3 $\rightarrow$ M3).
  • Figure 3: Ablation study of LeMoF components under different EHR backbones, with the ECG encoder fixed to ResNet. Performance is reported in terms of (a) Accuracy, (b) AUROC, and (c) F1-score for single-modality models and LeMoF variants (Only M1, Only M2, and Full).