MMSF: Multitask and Multimodal Supervised Framework for WSI Classification and Survival Analysis

TL;DR

MMSF introduces a linear-complexity multitask and multimodal framework for WSI classification and survival analysis that fuses patch-level graph context with instance-level clinical embeddings through a hierarchical fusion design. It replaces transformer-based MIL with a state space model-based Mamba backbone, enabling efficient processing of large patch sets (APS selects 512 patches) and delivering joint classification and survival predictions. The architecture combines a patch-level graph module, a clinical data embedding module, and an SE-based feature fusion mechanism to achieve state-of-the-art performance on CAMELYON16, TCGA-NSCLC, and five TCGA survival cohorts, with strong ablations validating each component. The approach offers clinically practical benefits through improved accuracy, interpretability via patch-score visualizations, and computational efficiency that supports deployment in real-world pathology workflows.

Abstract

Multimodal evidence is critical in computational pathology: gigapixel whole slide images capture tumor morphology, while patient-level clinical descriptors preserve complementary context for prognosis. Integrating such heterogeneous signals remains challenging because feature spaces exhibit distinct statistics and scales. We introduce MMSF, a multitask and multimodal supervised framework built on a linear-complexity MIL backbone that explicitly decomposes and fuses cross-modal information. MMSF comprises a graph feature extraction module embedding tissue topology at the patch level, a clinical data embedding module standardizing patient attributes, a feature fusion module aligning modality-shared and modality-specific representations, and a Mamba-based MIL encoder with multitask prediction heads. Experiments on CAMELYON16 and TCGA-NSCLC demonstrate 2.1--6.6\% accuracy and 2.2--6.9\% AUC improvements over competitive baselines, while evaluations on five TCGA survival cohorts yield 7.1--9.8\% C-index improvements compared with unimodal methods and 5.6--7.1\% over multimodal alternatives.

Figures (3)

• Figure 1: Overview of our proposed MMSF framework. (1) Patch-level Features Extraction. WSIs are cropped into patches and processed by foundation model UNI2 to extract 1536-dimensional patch features. (2) Patch-level Graph Construction. Spatial graphs are constructed based on spatial proximity and tissue similarity, processed by GNN, and aggregated via global average pooling to obtain graph embeddings. (3) Multimodal Features Fusion. Patch features and graph embeddings are fused early using our proposed feature fusion module (FFM), then processed by EfficientMIL with adaptive patch selector (APS) to obtain instance-level representations. And then, the instance-level representations are fused late with clinical embeddings using another FFM to obtain final features. (4) Instance-level Clinical Data Integration. Clinical data (age, gender, T stage, N stage, etc.) is processed separately through our proposed clinical data embedding module (CDE) into clinical embeddings. (5) Multitask Prediction. The final features are fed into task-specific prediction heads (survival head or classifier head) for final predictions.
• Figure 2: Visualization of patch scores from the adaptive patch selector (APS) on CAMELYON16 dataset for classification task. The heatmaps overlay patch scores $S_{patch}$ on WSI, where red regions (score $\approx$ 1.0) indicate highly informative patches selected by APS, and blue regions (score $\approx$ 0.0) represent less informative patches.
• Figure 3: Kaplan-Meier survival curves for high-risk and low-risk patient groups stratified by MMSF predictions across four TCGA datasets: TCGA-LUAD (top-left), TCGA-KIRC (top-right), TCGA-COAD (bottom-left), and TCGA-BLCA (bottom-right). Patients are divided into high-risk (red) and low-risk (green) groups based on predicted survival risk scores. Shaded regions represent 95% confidence intervals.