iMD4GC: Incomplete Multimodal Data Integration to Advance Precise Treatment Response Prediction and Survival Analysis for Gastric Cancer

TL;DR

iMD4GC tackles the challenge of incomplete multimodal data in gastric cancer by integrating clinical records, radiology, pathology, and genomics through unimodal attention, cross-modal interaction, and a novel more-to-fewer knowledge distillation strategy. The approach yields strong predictive performance for both neoadjuvant chemotherapy response (AUC up to 0.802 on GastricRes) and survival prognosis (C-index up to 0.714 on GastricSur and 0.661 on TCGA-STAD), outperforming a broad range of unimodal, multimodal, and missing-modality baselines. It further provides interpretability via Integrated Gradients and attention-based pathology localization, enabling insight into contributing clinical records, pathological patterns, radiology regions, and genomic biomarkers. The framework’s flexible fusion capability and scalability, along with its potential for biomarker discovery and extensibility to additional modalities, position it as a practical AI tool for precision oncology in gastric cancer.

Abstract

Gastric cancer (GC) is a prevalent malignancy worldwide, ranking as the fifth most common cancer with over 1 million new cases and 700 thousand deaths in 2020. Locally advanced gastric cancer (LAGC) accounts for approximately two-thirds of GC diagnoses, and neoadjuvant chemotherapy (NACT) has emerged as the standard treatment for LAGC. However, the effectiveness of NACT varies significantly among patients, with a considerable subset displaying treatment resistance. Ineffective NACT not only leads to adverse effects but also misses the optimal therapeutic window, resulting in lower survival rate. However, existing multimodal learning methods assume the availability of all modalities for each patient, which does not align with the reality of clinical practice. The limited availability of modalities for each patient would cause information loss, adversely affecting predictive accuracy. In this study, we propose an incomplete multimodal data integration framework for GC (iMD4GC) to address the challenges posed by incomplete multimodal data, enabling precise response prediction and survival analysis. Specifically, iMD4GC incorporates unimodal attention layers for each modality to capture intra-modal information. Subsequently, the cross-modal interaction layers explore potential inter-modal interactions and capture complementary information across modalities, thereby enabling information compensation for missing modalities. To evaluate iMD4GC, we collected three multimodal datasets for GC study: GastricRes (698 cases) for response prediction, GastricSur (801 cases) for survival analysis, and TCGA-STAD (400 cases) for survival analysis. The scale of our datasets is significantly larger than previous studies. The iMD4GC achieved impressive performance with an 80.2% AUC on GastricRes, 71.4% C-index on GastricSur, and 66.1% C-index on TCGA-STAD, significantly surpassing other compared methods.

Figures (9)

• Figure 1: Overview of treatment journey for GC patients, multimodal data acquisition in GC diagnosis and treatment, and incomplete multimodal data learning for GC. (A) A schematic workflow of gastric cancer treatment and prognosis, including NACT treatment, surgical resection, and survival analysis. (B) The multimodal data acquisition process in the diagnosis, treatment, and prognosis of GC, involving clinical records, pathology images, radiology images, and genomic profiles. (C) The pipeline of incomplete multimodal data integration framework for precise response prediction and survival analysis.
• Figure 2: ROC curves for response prediction, significance analysis for survival prediction, and comparative analysis for knowledge distillaiton. (A) ROC curve and AUC value of each fold on GastricRes dataset. The AUC values consistently exceed 80.0% across most folds, except for the third fold. (B-C) Kaplan-Meier curves on GastricSur and TCGA-STAD datasets: all patients are stratified into low-risk group and high-risk group according to the predicted risk scores. The tables provide additional information about the number of individuals at risk at each time point. (D-F) Performance comparison on three datasets before and after knowledge distillation.
• Figure 3: Contribution of individual clinical record for NACT response prediction and survival analysis. There are 14 clinical records involved in the GastricRes dataset and 28 clinical records involved in the TCGA-STAD dataset. The contribution values are calculated by Integrated Gradients sundararajan2017axiomatic. Positive contribution values signify a positive influence on the model prediction (response prediction and death risk), while negative values indicate a negative influence. Conversely, zero contribution values imply that the corresponding records have negligible impact on the model prediction.
• Figure 4: Heatmaps for pathological analysis. (A) Visualization comparison between good responders and non-responders. (B) Visualization comparison between low-risk and high-risk patients. The regions with high attention scores are deemed more valuable for model prediction, while those with low attention scores carry less significance. Right column shows the top-6 patches with the highest attention scores.
• Figure 5: Radiological analysis and genetic analysis. (A) An example of localization results for radiology images. The attention scores in iMD4GC provides the axial localization of the tumor region first. And then, utilizing Grad-CAM shows the sagittal and coronal localization of the tumor region. (B) The attention distribution of the top-100 genes for survival prediction. Each point represents the attention score of one sample. The table shows the top-20 genes with the highest contribution values.