Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PE-MVCNet: Multi-view and Cross-modal Fusion Network for Pulmonary Embolism Prediction

Zhaoxin Guo, Zhipeng Wang, Ruiquan Ge, Jianxun Yu, Feiwei Qin, Yuan Tian, Yuqing Peng, Yonghong Li, Changmiao Wang

TL;DR

The paper tackles PE prediction by fusing 3D CT angiography images with EMR data. It introduces PE-MVCNet, a three-branch architecture with an image-only path featuring multi-view self-attention, an EMR-only path, and a cross-modal CMAF fusion module to jointly leverage imaging and tabular data. On the SUMC dataset, the method achieves $AUROC=0.941$, $ACC=0.902$, and $F1=0.906$, outperforming single-modality baselines and prior multimodal approaches. These results demonstrate the value of cross-modal fusion and multi-view imaging for robust PE prediction and potential clinical deployment.

Abstract

The early detection of a pulmonary embolism (PE) is critical for enhancing patient survival rates. Both image-based and non-image-based features are of utmost importance in medical classification tasks. In a clinical setting, physicians tend to rely on the contextual information provided by Electronic Medical Records (EMR) to interpret medical imaging. However, very few models effectively integrate clinical information with imaging data. To address this shortcoming, we suggest a multimodal fusion methodology, termed PE-MVCNet, which capitalizes on Computed Tomography Pulmonary Angiography imaging and EMR data. This method comprises the Image-only module with an integrated multi-view block, the EMR-only module, and the Cross-modal Attention Fusion (CMAF) module. These modules cooperate to extract comprehensive features that subsequently generate predictions for PE. We conducted experiments using the publicly accessible Stanford University Medical Center dataset, achieving an AUROC of 94.1%, an accuracy rate of 90.2%, and an F1 score of 90.6%. Our proposed model outperforms existing methodologies, corroborating that our multimodal fusion model excels compared to models that use a single data modality. Our source code is available at https://github.com/LeavingStarW/PE-MVCNET.

PE-MVCNet: Multi-view and Cross-modal Fusion Network for Pulmonary Embolism Prediction

TL;DR

The paper tackles PE prediction by fusing 3D CT angiography images with EMR data. It introduces PE-MVCNet, a three-branch architecture with an image-only path featuring multi-view self-attention, an EMR-only path, and a cross-modal CMAF fusion module to jointly leverage imaging and tabular data. On the SUMC dataset, the method achieves , , and , outperforming single-modality baselines and prior multimodal approaches. These results demonstrate the value of cross-modal fusion and multi-view imaging for robust PE prediction and potential clinical deployment.

Abstract

The early detection of a pulmonary embolism (PE) is critical for enhancing patient survival rates. Both image-based and non-image-based features are of utmost importance in medical classification tasks. In a clinical setting, physicians tend to rely on the contextual information provided by Electronic Medical Records (EMR) to interpret medical imaging. However, very few models effectively integrate clinical information with imaging data. To address this shortcoming, we suggest a multimodal fusion methodology, termed PE-MVCNet, which capitalizes on Computed Tomography Pulmonary Angiography imaging and EMR data. This method comprises the Image-only module with an integrated multi-view block, the EMR-only module, and the Cross-modal Attention Fusion (CMAF) module. These modules cooperate to extract comprehensive features that subsequently generate predictions for PE. We conducted experiments using the publicly accessible Stanford University Medical Center dataset, achieving an AUROC of 94.1%, an accuracy rate of 90.2%, and an F1 score of 90.6%. Our proposed model outperforms existing methodologies, corroborating that our multimodal fusion model excels compared to models that use a single data modality. Our source code is available at https://github.com/LeavingStarW/PE-MVCNET.
Paper Structure (13 sections, 2 equations, 2 figures, 2 tables)

This paper contains 13 sections, 2 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Overview of the architecture
  4. Image-only process module
  5. EMR-only process module
  6. Cross-modal fusion module
  7. Experiment
  8. Experimental setting
  9. Comparison with state-of-the-art models
  10. Ablation study
  11. Conclusion
  12. Compliance with Ethical Standards
  13. Acknowledgements

Figures (2)

  • Figure 1: The overall framework of the proposed PE-MVCNet model for PE prediction. The model comprises the Image-only module, EMR-only module, and Cross-modal Attention Fusion (CMAF) module. The Image-only model employs spatial and dimensional attention to investigate dependency relationships on spatial, channel, and dimensional aspects, respectively. Conversely, the CMAF module is designed to capture the correlation between image and tabular features.
  • Figure 2: Multi-View Coupled Self-Attention Block. 'DA' denotes Dimensional Attention, and 'SA' signifies Spatial Attention.