Enhancing Incomplete Multi-modal Brain Tumor Segmentation with Intra-modal Asymmetry and Inter-modal Dependency

TL;DR

A novel approach that enhances the BTS model from two perspectives is proposed, introducing a pre-training stage that generates a diverse pre-training dataset covering a wide range of different combinations of tumor shapes and brain anatomy and a post-training stage that enables the model to reconstruct missing modalities in the prediction results when only partial modalities are available.

Abstract

Deep learning-based brain tumor segmentation (BTS) models for multi-modal MRI images have seen significant advancements in recent years. However, a common problem in practice is the unavailability of some modalities due to varying scanning protocols and patient conditions, making segmentation from incomplete MRI modalities a challenging issue. Previous methods have attempted to address this by fusing accessible multi-modal features, leveraging attention mechanisms, and synthesizing missing modalities using generative models. However, these methods ignore the intrinsic problems of medical image segmentation, such as the limited availability of training samples, particularly for cases with tumors. Furthermore, these methods require training and deploying a specific model for each subset of missing modalities. To address these issues, we propose a novel approach that enhances the BTS model from two perspectives. Firstly, we introduce a pre-training stage that generates a diverse pre-training dataset covering a wide range of different combinations of tumor shapes and brain anatomy. Secondly, we propose a post-training stage that enables the model to reconstruct missing modalities in the prediction results when only partial modalities are available. To achieve the pre-training stage, we conceptually decouple the MRI image into two parts: `anatomy' and `tumor'. We pre-train the BTS model using synthesized data generated from the anatomy and tumor parts across different training samples. ... Extensive experiments demonstrate that our proposed method significantly improves the performance over the baseline and achieves new state-of-the-art results on three brain tumor segmentation datasets: BRATS2020, BRATS2018, and BRATS2015.

Figures (8)

• Figure 1: From left to right: MRI images for four different modalities (Flair, T1ce, T1, and T2) along with their corresponding labels.
• Figure 2: Overview of the proposed approach. 'Seg' is short for 'segmentation'. (a) Pre-train the model from scratch with the synthetic data by exploiting the asymmetry of brain MRI images. (b) Conduct standard training with real training samples based on the pre-trained model. (c) Conduct distillation-based post-training to improve the model's ability to deal with partial-modalitty inputs.
• Figure 3: Proposed sample synthesis process for generating the pre-training dataset.
• Figure 4: The synthetic samples. The tumor part of $X_B$ is embedded into $X_A$, which generates $X_{AB}$. The ground truth $Y_{AB}$ is generated by the fusion of $Y_A$ and $Y_B$. For each modality, we provide one example (Flair, T1ce, T1, and T2 are listed from top to bottom).
• Figure 5: The visualization of segmentation results on BRATS2020 between our method and state-of-the-art methods. From left to right, we demonstrate the MRI images for the Flair modality, T1ce modality, T1 modality, and T2 modality. Following the ground truth, the prediction of U-HVEDdorent2019hetero, RobustSegchen2019robust, RFNetding2021rfnet and our prediction.