RobSurv: Vector Quantization-Based Multi-Modal Learning for Robust Cancer Survival Prediction

TL;DR

RobSurv tackles robust cancer survival prediction from multi-modal CT and PET imaging by introducing DualVQ, a dual-path vector quantization framework, and DualPatchFuse, a patch-wise cross-modal fusion mechanism. The method learns complementary discrete and continuous representations, preserving local structure while leveraging global context through Transformer-based fusion, and optimizes a hybrid survival objective with competing risks. Empirical results on three datasets (HECKTOR, H&N1, NSCLC Radiogenomics) show state-of-the-art concordance indices and strong robustness to noise, with only small degradations under severe noise relative to baselines. The approach demonstrates strong generalization across cancer types and imaging protocols, suggesting practical utility for reliable clinical prognosis and treatment planning.

Abstract

Cancer survival prediction using multi-modal medical imaging presents a critical challenge in oncology, mainly due to the vulnerability of deep learning models to noise and protocol variations across imaging centers. Current approaches struggle to extract consistent features from heterogeneous CT and PET images, limiting their clinical applicability. We address these challenges by introducing RobSurv, a robust deep-learning framework that leverages vector quantization for resilient multi-modal feature learning. The key innovation of our approach lies in its dual-path architecture: one path maps continuous imaging features to learned discrete codebooks for noise-resistant representation, while the parallel path preserves fine-grained details through continuous feature processing. This dual representation is integrated through a novel patch-wise fusion mechanism that maintains local spatial relationships while capturing global context via Transformer-based processing. In extensive evaluations across three diverse datasets (HECKTOR, H\&N1, and NSCLC Radiogenomics), RobSurv demonstrates superior performance, achieving concordance index of 0.771, 0.742, and 0.734 respectively - significantly outperforming existing methods. Most notably, our model maintains robust performance even under severe noise conditions, with performance degradation of only 3.8-4.5\% compared to 8-12\% in baseline methods. These results, combined with strong generalization across different cancer types and imaging protocols, establish RobSurv as a promising solution for reliable clinical prognosis that can enhance treatment planning and patient care.

Figures (3)

• Figure 1: Architecture of the proposed RobSurv architecture for robust survival prediction, consisting of three key components: Dual Vector Quantization (DualVQ) for parallel CT-PET processing, DualPatchFuse for cross-modal attention and fusion, and a hybrid survival prediction network for patient outcome estimation with competing risk analysis.
• Figure 2: Box plot visualization of $C_{td}$-index distributions across varying proportions of noisy samples (10%-100%) for NSCLC, HECKTOR, and H&N1 datasets, demonstrating the impact of noise on model robustness.
• Figure 3: Kaplan-Meier survival curves demonstrating risk stratification performance. The plot on top shows the survival analysis on clean samples, while the plot at the bottom illustrates the model's performance when subjected to noisy data conditions.