Embedding-based Multimodal Learning on Pan-Squamous Cell Carcinomas for Improved Survival Outcomes

TL;DR

This work proposes PARADIGM, a Graph Neural Network (GNN) framework that learns from multimodal, heterogeneous datasets to improve clinical outcome prediction, offering insights on heterogeneous data integration and the benefits of converging maximum data views.

Abstract

Cancer clinics capture disease data at various scales, from genetic to organ level. Current bioinformatic methods struggle to handle the heterogeneous nature of this data, especially with missing modalities. We propose PARADIGM, a Graph Neural Network (GNN) framework that learns from multimodal, heterogeneous datasets to improve clinical outcome prediction. PARADIGM generates embeddings from multi-resolution data using foundation models, aggregates them into patient-level representations, fuses them into a unified graph, and enhances performance for tasks like survival analysis. We train GNNs on pan-Squamous Cell Carcinomas and validate our approach on Moffitt Cancer Center lung SCC data. Multimodal GNN outperforms other models in patient survival prediction. Converging individual data modalities across varying scales provides a more insightful disease view. Our solution aims to understand the patient's circumstances comprehensively, offering insights on heterogeneous data integration and the benefits of converging maximum data views.

Figures (11)

• Figure 1: The schematic layout of the multi-modal data integration framework is presented. The framework integrates multi-omics data, pathology reports, clinical/EHR data, and histopathology imaging data to generate sample embeddings using modality-specific foundation models. These embeddings are pooled and aggregated in patient embeddings which are then used to construct patient graphs. These patient graphs are analyzed using Graph Neural Network (GNN) to predict overall survival outcome.
• Figure 2: A schematic layout of the multimodal oncology database system (MINDS) MINDS to collate, retrieve, harmonize, and serve multimodal data to PARADIGM models for training. We integrate data from National Cancer Institute (NCI)'s publicly available resources including, Cancer Research Data Commons (CRDC), Genomic Data Commons (GDC), Imaging Data Commons (IDC), and Proteomic Data Commons (PDC) CRDCCPTACHCMITCGA.
• Figure 3: C-Index on training different models on the SCC data. The multimodal datasets consist of clinical, pathology reports, whole slide images (WSIs), and molecular data.
• Figure 4: C-Index for OS predictions from different models on the SCC data comprising clinical, pathology reports, and WSIs data.
• Figure 5: C-Index for OS predictions from different models on the SCC data comprising clinical, molecular, and WSIs.