Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TMI-CLNet: Triple-Modal Interaction Network for Chronic Liver Disease Prognosis From Imaging, Clinical, and Radiomic Data Fusion

Linglong Wu, Xuhao Shan, Ruiquan Ge, Ruoyu Liang, Chi Zhang, Yonghong Li, Ahmed Elazab, Huoling Luo, Yunbi Liu, Changmiao Wang

TL;DR

Chronic liver disease prognosis benefits from multimodal data, but integrating CT imaging, radiomic features, and clinical information is challenging due to modality heterogeneity. The authors propose TMI-CLNet, a triple-modal network featuring Intra-Modal Aggregation (IMA) to refine each modality, Triple-Modal Cross-Attention Fusion (TCAF) to extract cross-modal interactions, and a Triple-Modal Feature Fusion (TMFF) loss to align representations across modalities. On a private cohort of 184 patients, TMI-CLNet with 5-fold cross-validation outperforms unimodal and other multimodal baselines, with accuracy 83.12% and AUC 0.8223, and ablations confirm the contributions of each module. The approach demonstrates strong potential for improved, data-driven prognosis in chronic liver disease and can be extended to additional diseases or modalities with similar heterogeneity challenges.

Abstract

Chronic liver disease represents a significant health challenge worldwide and accurate prognostic evaluations are essential for personalized treatment plans. Recent evidence suggests that integrating multimodal data, such as computed tomography imaging, radiomic features, and clinical information, can provide more comprehensive prognostic information. However, modalities have an inherent heterogeneity, and incorporating additional modalities may exacerbate the challenges of heterogeneous data fusion. Moreover, existing multimodal fusion methods often struggle to adapt to richer medical modalities, making it difficult to capture inter-modal relationships. To overcome these limitations, We present the Triple-Modal Interaction Chronic Liver Network (TMI-CLNet). Specifically, we develop an Intra-Modality Aggregation module and a Triple-Modal Cross-Attention Fusion module, which are designed to eliminate intra-modality redundancy and extract cross-modal information, respectively. Furthermore, we design a Triple-Modal Feature Fusion loss function to align feature representations across modalities. Extensive experiments on the liver prognosis dataset demonstrate that our approach significantly outperforms existing state-of-the-art unimodal models and other multi-modal techniques. Our code is available at https://github.com/Mysterwll/liver.git.

TMI-CLNet: Triple-Modal Interaction Network for Chronic Liver Disease Prognosis From Imaging, Clinical, and Radiomic Data Fusion

TL;DR

Chronic liver disease prognosis benefits from multimodal data, but integrating CT imaging, radiomic features, and clinical information is challenging due to modality heterogeneity. The authors propose TMI-CLNet, a triple-modal network featuring Intra-Modal Aggregation (IMA) to refine each modality, Triple-Modal Cross-Attention Fusion (TCAF) to extract cross-modal interactions, and a Triple-Modal Feature Fusion (TMFF) loss to align representations across modalities. On a private cohort of 184 patients, TMI-CLNet with 5-fold cross-validation outperforms unimodal and other multimodal baselines, with accuracy 83.12% and AUC 0.8223, and ablations confirm the contributions of each module. The approach demonstrates strong potential for improved, data-driven prognosis in chronic liver disease and can be extended to additional diseases or modalities with similar heterogeneity challenges.

Abstract

Chronic liver disease represents a significant health challenge worldwide and accurate prognostic evaluations are essential for personalized treatment plans. Recent evidence suggests that integrating multimodal data, such as computed tomography imaging, radiomic features, and clinical information, can provide more comprehensive prognostic information. However, modalities have an inherent heterogeneity, and incorporating additional modalities may exacerbate the challenges of heterogeneous data fusion. Moreover, existing multimodal fusion methods often struggle to adapt to richer medical modalities, making it difficult to capture inter-modal relationships. To overcome these limitations, We present the Triple-Modal Interaction Chronic Liver Network (TMI-CLNet). Specifically, we develop an Intra-Modality Aggregation module and a Triple-Modal Cross-Attention Fusion module, which are designed to eliminate intra-modality redundancy and extract cross-modal information, respectively. Furthermore, we design a Triple-Modal Feature Fusion loss function to align feature representations across modalities. Extensive experiments on the liver prognosis dataset demonstrate that our approach significantly outperforms existing state-of-the-art unimodal models and other multi-modal techniques. Our code is available at https://github.com/Mysterwll/liver.git.

Paper Structure

This paper contains 11 sections, 6 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Multi-modal Interaction Module
  4. Triple-modal Cross-Attention Fusion Module
  5. Loss Function
  6. Experiments
  7. Experimental Setup
  8. Experimental Results
  9. Conclusion
  10. COMPLIANCE WITH ETHICAL STANDARDS
  11. ACKNOWLEDGMENTS

Figures (3)

  • Figure 1: Overview of our proposed TMI-CLNet. Given the CT scans, radiomic features, and clinical text, we feed them into an independent modal encoder to extract high-dimensional features. Subsequently, these features are fused through multi-modal interaction to incorporate global characteristics for prognosis classification. Simultaneously, the TMFF loss is employed to align multi-modal features.
  • Figure 2: Overview of our proposed TCAF module. The highlighting the process by the independent features from Modality 1 are integrated to produce cross-modal features.
  • Figure 3: Visual examples illustrating the model's focus before and after fusion. The redder areas indicate higher model attention. Red boxes highlight the liver region, whereas yellow boxes denote areas deemed irrelevant by the model.