Decouple, Reorganize, and Fuse: A Multimodal Framework for Cancer Survival Prediction

TL;DR

A novel Decoupling-Reorganization-Fusion framework (DeReF), which devises a random feature reorganization strategy between modalities decoupling and dynamic MoE fusion modules that increases the diversity of feature combinations and granularity, enhancing the generalization ability of the subsequent expert networks.

Abstract

Cancer survival analysis commonly integrates information across diverse medical modalities to make survival-time predictions. Existing methods primarily focus on extracting different decoupled features of modalities and performing fusion operations such as concatenation, attention, and \revm{Mixture-of-Experts (MoE)-based fusion. However, these methods still face two key challenges: i) Fixed fusion schemes (concatenation and attention) can lead to model over-reliance on predefined feature combinations, limiting the dynamic fusion of decoupled features; ii) in MoE-based fusion methods, each expert network handles separate decoupled features, which limits information interaction among the decoupled features. To address these challenges, we propose a novel Decoupling-Reorganization-Fusion framework (DeReF), which devises a random feature reorganization strategy between modalities decoupling and dynamic MoE fusion modules.Its advantages are: i) it increases the diversity of feature combinations and granularity, enhancing the generalization ability of the subsequent expert networks; ii) it overcomes the problem of information closure and helps expert networks better capture information among decoupled features. Additionally, we incorporate a regional cross-attention network within the modality decoupling module to improve the representation quality of decoupled features. Extensive experimental results on our in-house Liver Cancer (LC) and three widely used TCGA public datasets confirm the effectiveness of our proposed method. Codes are available at https://github.com/ZJUMAI/DeReF.

Figures (8)

• Figure 1: Model Architecture Diagram. The random reorganization of decoupled features is conducive to 1) increasing the diversity of feature combinations and granularity, enhancing the generalization performance of expert networks, and 2) increasing the interaction of different decoupled features in expert networks.
• Figure 2: An overview of our proposed framework. It consists of four modules: the Feature Extraction Module, Feature Decoupling Module, Dynamic MoE Fusion Module (containing Random Feature Reorganization and an MoE model), and Survival Prediction Module.
• Figure 3: Illustration of Feature Reorganization Module. The reorganized decoupled features are fed into each of the four subsequent expert networks.
• Figure 4: Visualization results of decoupled features. In each sub-figure, the left part displays t-SNE visualization of the decoupled features, and the right part shows the heatmap, which displays the centered kernel alignment (CKA) similarity of each pair-wise decoupled features.
• Figure 5: Visualization results of dynamic average weights from the output of MoE Gating unit.