MMMNA-Net for Overall Survival Time Prediction of Brain Tumor Patients

TL;DR

This paper tackles glioma overall survival (OS) time prediction using four MRI modalities (T1, T1Ce, T2, FLAIR) and introduces MMMNA-Net, a multi-modal, multi-channel, multi-scale framework with an improved non-local attention-based fusion that operates across scales. The method combines a compact 3D ResNet18-based backbone with a Linformer-inspired, dimension-reduced self-attention mechanism to fuse features from all modalities, and employs branch-specific weighted pooling and per-modality focal losses plus a fusion loss weighted by $\lambda$ to guide joint learning. On BraTS2020 data, MMMNA-Net achieves state-of-the-art accuracy ($0.6989$) and F-score ($0.6613$), outperforming several baselines including MMNet, and demonstrates robustness to missing modalities by substituting unavailable inputs with FLAIR. The approach uses 10-fold cross-validation and demonstrates practical applicability in real-world settings where not all modalities may be available, with code released for reproducibility.

Abstract

Overall survival (OS) time is one of the most important evaluation indices for gliomas situations. Multimodal Magnetic Resonance Imaging (MRI) scans play an important role in the study of glioma prognosis OS time. Several deep learning-based methods are proposed for the OS time prediction on multi-modal MRI problems. However, these methods usually fuse multi-modal information at the beginning or at the end of the deep learning networks and lack the fusion of features from different scales. In addition, the fusion at the end of networks always adapts global with global (eg. fully connected after concatenation of global average pooling output) or local with local (eg. bilinear pooling), which loses the information of local with global. In this paper, we propose a novel method for multi-modal OS time prediction of brain tumor patients, which contains an improved nonlocal features fusion module introduced on different scales. Our method obtains a relative 8.76% improvement over the current state-of-art method (0.6989 vs. 0.6426 on accuracy). Extensive testing demonstrates that our method could adapt to situations with missing modalities. The code is available at https://github.com/TangWen920812/mmmna-net.

Figures (3)

• Figure 1: Overall structure of Multi-modal Multi-channel Multi-scale Non-local Attention Network.
• Figure 2: Non-local multi-modal fusion Flowchart
• Figure 3: Multi-scale features fusion module